Introduction
I have written extensively about experimental testing forย the purpose of material model calibration.ย I recently realized, however, that I have not analyzed or quantified how much you can expect the experimental accuracy to improve if you run multiple repeated experiments. I usually recommended using 2 or 3 repeated tests for each experiment, without any real justification. In this article I will show how you can determine how much the experimental accuracy is likely to improve!
Background
When you run multiple repeated experiments you will not get exactly the same results each time. Instead it is reasonable to assume that the stress value for each strain point will be normally distributed with an mean and a standard deviation. Typically you do not know much about this distribution. So if you only run one experiment, than you have 68% chance that the obtained result is within one standard deviation away from the mean. This is the reason for running multiple repeated experiments. The average of those experiments is more likely to be close to the true mean value.ย In this article I will calculate just how much one can expect the results to improve.
Case Study 1
In this study I will assume that the stress is normalized, so that the experimental mean stress has a mean value of 1.0.ย I will also assume that the standard deviation of the experimental stress distribution is 0.1. I will then use Python to generate 100,000 stress values from the assumed experimental normal distribution of stresses. Based on those values I will then calculate the expected mean absolute error in the measured stress under the assumption that one once experiment was perform. The Python code I used is shown below.
import numpy as np
stress_real = 1.0
std = 0.1
N = 100000
print(f"The actual value of the stress is {stress_real}")
print(f"The standard deviation in the experiments is assumed to be {std}\n")
print(f"Perform {N} experiments.")
stress_exp = np.random.normal(stress_real, std, N)
stress_exp_average = np.mean(stress_exp)
print(f"Average experimental value = {stress_exp_average:.6f}")
stress_exp_std = np.std(stress_exp)
print(f"Standard deviation of experiments = {stress_exp_std:.6f}")
stress_exp_err = np.mean(np.abs(stress_exp - stress_real))
print(f"Average mean absolute error = {stress_exp_err:.6f}")
The results from running this code is shown below. In summary, if your experimental tests have a standard deviation that is 10% of the real stress, and you only run one experiment, then you expect an error that is about 8.0%.
The actual value of the stress is 1.0
Perform 100000 experiments.
The standard deviation in the experiments is assumed to be 0.1
Average experimental value = 1.002418
Standard deviation of experiments = 0.103060
Average mean absolute error = 0.083809
Case Study 2
This study is the same as Study 1, except that I will take the average of two experiments. Here’s the code:
import numpy as np
stress_real = 1.0
std = 0.1
N = 100000
print(f"The actual value of the stress is {stress_real}")
print(f"The standard deviation in the experiments is assumed to be {std}\n")
print(f"Perform {N} experiments twice.")
stress_exp_1 = np.random.normal(stress_real, std, N)
stress_exp_2 = np.random.normal(stress_real, std, N)
stress_exp_average_1 = np.mean(stress_exp_1)
stress_exp_average_2 = np.mean(stress_exp_2)
print(f"Average experimental value (test 1) = {stress_exp_average_1:.6f}")
print(f"Average experimental value (test 2) = {stress_exp_average_2:.6f}")
stress_exp_std_1 = np.std(stress_exp_1)
stress_exp_std_2 = np.std(stress_exp_2)
print(f"Standard deviation of experiments (test 1) = {stress_exp_std_1:.6f}")
print(f"Standard deviation of experiments (test 2) = {stress_exp_std_2:.6f}")
stress_exp = 0.5 * (stress_exp_1 + stress_exp_2)
stress_exp_err = np.mean(np.abs(stress_exp - stress_real))
print(f"Average mean absolute error = {stress_exp_err:.6f}")
The results from running this code is shown below. In summary, if your experimental tests have a standard deviation that is 10% of the real stress, and you use the average of 2 experiments, then you expect an error that is about 5.6%.
The actual value of the stress is 1.0
The standard deviation in the experiments is assumed to be 0.1
Perform 100000 experiments twice.
Average experimental value (test 1) = 0.998783
Average experimental value (test 2) = 1.000566
Standard deviation of experiments (test 1) = 0.098807
Standard deviation of experiments (test 2) = 0.099501
Average mean absolute error = 0.056032
Case Study 3
This study is the same as Study 2, except that I will take the average of three experiments. Here’s the code:
import numpy as np
stress_real = 1.0
std = 0.1
N = 100000
print(f"The actual value of the stress is {stress_real}")
print(f"The standard deviation in the experiments is assumed to be {std}\n")
print(f"Perform {N} experiments three times.")
stress_exp_1 = np.random.normal(stress_real, std, N)
stress_exp_2 = np.random.normal(stress_real, std, N)
stress_exp_3 = np.random.normal(stress_real, std, N)
stress_exp_average_1 = np.mean(stress_exp_1)
stress_exp_average_2 = np.mean(stress_exp_2)
stress_exp_average_3 = np.mean(stress_exp_3)
print(f"Average experimental value (test 1) = {stress_exp_average_1:.6f}")
print(f"Average experimental value (test 2) = {stress_exp_average_2:.6f}")
print(f"Average experimental value (test 3) = {stress_exp_average_3:.6f}")
stress_exp_std_1 = np.std(stress_exp_1)
stress_exp_std_2 = np.std(stress_exp_2)
stress_exp_std_3 = np.std(stress_exp_3)
print(f"Standard deviation of experiments (test 1) = {stress_exp_std_1:.6f}")
print(f"Standard deviation of experiments (test 2) = {stress_exp_std_2:.6f}")
print(f"Standard deviation of experiments (test 3) = {stress_exp_std_3:.6f}")
stress_exp = (stress_exp_1 + stress_exp_2 + stress_exp_3) / 3.0
stress_exp_err = np.mean(np.abs(stress_exp - stress_real))
print(f"Average mean absolute error = {stress_exp_err:.6f}")
The results from running this code is shown below. In summary, if your experimental tests have a standard deviation that is 10% of the real stress, and you use the average of 3 experiments, then you expect an error that is about 4.6%.
The actual value of the stress is 1.0
The standard deviation in the experiments is assumed to be 0.1
Perform 100000 experiments three times.
Average experimental value (test 1) = 1.000048
Average experimental value (test 2) = 0.999853
Average experimental value (test 3) = 0.999961
Standard deviation of experiments (test 1) = 0.100313
Standard deviation of experiments (test 2) = 0.100074
Standard deviation of experiments (test 3) = 0.100143
Average mean absolute error = 0.046049
Case Study 4
This study is the same as Study 3, except that I will take the median of three experiments. Here’s the code:
import numpy as np
stress_real = 1.0
std = 0.1
N = 100000
print(f"The actual value of the stress is {stress_real}")
print(f"The standard deviation in the experiments is assumed to be {std}\n")
print(f"Perform {N} experiments three times.")
stress_exp_1 = np.random.normal(stress_real, std, N)
stress_exp_2 = np.random.normal(stress_real, std, N)
stress_exp_3 = np.random.normal(stress_real, std, N)
stress_exp_average_1 = np.mean(stress_exp_1)
stress_exp_average_2 = np.mean(stress_exp_2)
stress_exp_average_3 = np.mean(stress_exp_3)
print(f"Average experimental value (test 1) = {stress_exp_average_1:.6f}")
print(f"Average experimental value (test 2) = {stress_exp_average_2:.6f}")
print(f"Average experimental value (test 3) = {stress_exp_average_3:.6f}")
stress_exp_std_1 = np.std(stress_exp_1)
stress_exp_std_2 = np.std(stress_exp_2)
stress_exp_std_3 = np.std(stress_exp_3)
print(f"Standard deviation of experiments (test 1) = {stress_exp_std_1:.6f}")
print(f"Standard deviation of experiments (test 2) = {stress_exp_std_2:.6f}")
print(f"Standard deviation of experiments (test 3) = {stress_exp_std_3:.6f}")
stress_exp = np.zeros(N)
for i in range(N):
stress_exp[i] = np.median([stress_exp_1[i], stress_exp_2[i], stress_exp_3[i]])
stress_exp_err = np.mean(np.abs(stress_exp - stress_real))
print(f"Average mean absolute error = {stress_exp_err:.6f}")
The results from running this code is shown below. In summary, if your experimental tests have a standard deviation that is 10% of the real stress, and you use the median of 3 experiments, then you expect an error that is about 5.3%.
The actual value of the stress is 1.0
The standard deviation in the experiments is assumed to be 0.1
Perform 100000 experiments three times.
Average experimental value (test 1) = 1.000220
Average experimental value (test 2) = 0.999834
Average experimental value (test 3) = 1.000290
Standard deviation of experiments (test 1) = 0.099537
Standard deviation of experiments (test 2) = 0.100173
Standard deviation of experiments (test 3) = 0.100027
Average mean absolute error = 0.053325
Case Study 5
This study is the same as Study 3, except that I will take the average of five experiments. Here’s the code:
import numpy as np
stress_real = 1.0
std = 0.1
N = 100000
print(f"The actual value of the stress is {stress_real}")
print(f"The standard deviation in the experiments is assumed to be {std}\n")
print(f"Perform {N} experiments 5 times.")
stress_exp_1 = np.random.normal(stress_real, std, N)
stress_exp_2 = np.random.normal(stress_real, std, N)
stress_exp_3 = np.random.normal(stress_real, std, N)
stress_exp_4 = np.random.normal(stress_real, std, N)
stress_exp_5 = np.random.normal(stress_real, std, N)
stress_exp_average_1 = np.mean(stress_exp_1)
stress_exp_average_2 = np.mean(stress_exp_2)
stress_exp_average_3 = np.mean(stress_exp_3)
stress_exp_average_4 = np.mean(stress_exp_4)
stress_exp_average_5 = np.mean(stress_exp_5)
print(f"Average experimental value (test 1) = {stress_exp_average_1:.6f}")
print(f"Average experimental value (test 2) = {stress_exp_average_2:.6f}")
print(f"Average experimental value (test 3) = {stress_exp_average_3:.6f}")
print(f"Average experimental value (test 4) = {stress_exp_average_4:.6f}")
print(f"Average experimental value (test 5) = {stress_exp_average_5:.6f}")
stress_exp_std_1 = np.std(stress_exp_1)
stress_exp_std_2 = np.std(stress_exp_2)
stress_exp_std_3 = np.std(stress_exp_3)
stress_exp_std_4 = np.std(stress_exp_4)
stress_exp_std_5 = np.std(stress_exp_5)
print(f"Standard deviation of experiments (test 1) = {stress_exp_std_1:.6f}")
print(f"Standard deviation of experiments (test 2) = {stress_exp_std_2:.6f}")
print(f"Standard deviation of experiments (test 3) = {stress_exp_std_3:.6f}")
print(f"Standard deviation of experiments (test 4) = {stress_exp_std_4:.6f}")
print(f"Standard deviation of experiments (test 5) = {stress_exp_std_5:.6f}")
stress_exp = (stress_exp_1 + stress_exp_2 + stress_exp_3 + stress_exp_4 + stress_exp_5) / 5.0
stress_exp_err = np.mean(np.abs(stress_exp - stress_real))
print(f"Average mean absolute error = {stress_exp_err:.6f}")
The results from running this code is shown below. In summary, if your experimental tests have a standard deviation that is 10% of the real stress, and you use the average of 5 experiments, then you expect an error that is about 3.5%.
The actual value of the stress is 1.0
The standard deviation in the experiments is assumed to be 0.1
Perform 100000 experiments 5 times.
Average experimental value (test 1) = 0.999901
Average experimental value (test 2) = 1.000032
Average experimental value (test 3) = 0.999821
Average experimental value (test 4) = 0.999931
Average experimental value (test 5) = 1.000180
Standard deviation of experiments (test 1) = 0.099850
Standard deviation of experiments (test 2) = 0.100129
Standard deviation of experiments (test 3) = 0.100320
Standard deviation of experiments (test 4) = 0.099909
Standard deviation of experiments (test 5) = 0.099568
Average mean absolute error = 0.035809
Summary
Based on the results above, we can conclude the following:
Taking the average of 2 experiments reduces the experimental error by 30%
Taking the average of 3 experiments reduces the experimental error by 43%
Taking the average of 5 experiments reduces the experimental error by 56%
- Using the median value of repeated experiments is less accurate than taking the average.
