Accuracy of Quantum Chemistry ML modelsCan quantum entanglement affect the chemistry of molecules?Mathematics...
Is right click on tables bad UX
Manager told a colleague of mine I was getting fired soon
How to refresh wired service getRecord manually?
Is there an in-universe explanation of how Frodo's arrival in Valinor was recorded in the Red Book?
The answer is a girl's name (my future granddaughter) - can anyone help?
Why did they use ultrafast diodes in a 50 or 60 Hz bridge?
If I travelled back in time to invest in X company to make a fortune, roughly what is the probability that it would fail?
Is it appropriate to "shop" through high-impact journals before sending the paper to more specialized journals?
Airport Security - advanced check, 4th amendment breach
How to protect bash function from being overridden?
Booting Ubuntu from USB drive on MSI motherboard -- EVERYTHING fails
Can my Beast Master ranger's baboon animal companion use her Wand of Magic Missiles?
Present participles of the verb esse
Citing CPLEX 12.9
Isn't the detector always measuring, and thus always collapsing the state?
Is "Ram married his daughter" ambiguous?
Is it possible for both sides of an encounter to be surprised?
Did Joe Biden "stop a prosecution" into his son in Ukraine? And did he brag about stopping the prosecution?
What action is recommended if your accommodation refuses to let you leave without paying additional fees?
Is the Basilisk Jaw a Slayer only drop?
Looking for circuit board material that can be dissolved
Why is music is taught by reading sheet music?
What did the Federation give the Prophets in exchange for access to the wormhole in DS9?
Is there a pattern for handling conflicting function parameters?
Accuracy of Quantum Chemistry ML models
Can quantum entanglement affect the chemistry of molecules?Mathematics in inorganic and quantum chemistryManifolds in quantum chemistryQuantum Chemistry: Small imaginary frequenciesCorrelation energy in quantum chemistryPerturbation theory in Quantum/Computational ChemistryCounting basis sets in quantum chemistry
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{
margin-bottom:0;
}
$begingroup$
I am trying to compare the performance of few Quantum Chemistry property prediction ML models. I was looking at the following table from DOI:
10.1039/c7sc02664a
The problem is that it does not contain any units. A preprint (https://arxiv.org/pdf/1909.00259.pdf, Table 4) suggests that the energy units are Hartree. Among energy properties, HOMO, LUMO, and gap have reasonable values for Hartree unit. However, values reported for U0, U, H, and G look absurd. As a comparison, look at MAE for mean value of each property:
These are in [eV] units (1 Ha = 27.212 eV). In other words, for instance MPNN models MAE for U property is 55.78 [eV] vs 8.25 [eV] for the mean baseline. This does not make sense at all. [k cal/mol] sounds more reasonable but then why half the energies in one unit the other half in other unit, and then why there are no units in a paper published by Royal Society of Chemistry. Seems like I'm missing something. Thanks!
quantum-chemistry machine-learning
asked 9 hours ago
BladeBlade
1306 bronze badges
$endgroup$
add a comment
|
$begingroup$
I am trying to compare the performance of few Quantum Chemistry property prediction ML models. I was looking at the following table from DOI:
10.1039/c7sc02664a
The problem is that it does not contain any units. A preprint (https://arxiv.org/pdf/1909.00259.pdf, Table 4) suggests that the energy units are Hartree. Among energy properties, HOMO, LUMO, and gap have reasonable values for Hartree unit. However, values reported for U0, U, H, and G look absurd. As a comparison, look at MAE for mean value of each property:
These are in [eV] units (1 Ha = 27.212 eV). In other words, for instance MPNN models MAE for U property is 55.78 [eV] vs 8.25 [eV] for the mean baseline. This does not make sense at all. [k cal/mol] sounds more reasonable but then why half the energies in one unit the other half in other unit, and then why there are no units in a paper published by Royal Society of Chemistry. Seems like I'm missing something. Thanks!
quantum-chemistry machine-learning
asked 9 hours ago
BladeBlade
1306 bronze badges
$endgroup$
add a comment
|
$begingroup$
I am trying to compare the performance of few Quantum Chemistry property prediction ML models. I was looking at the following table from DOI:
10.1039/c7sc02664a
The problem is that it does not contain any units. A preprint (https://arxiv.org/pdf/1909.00259.pdf, Table 4) suggests that the energy units are Hartree. Among energy properties, HOMO, LUMO, and gap have reasonable values for Hartree unit. However, values reported for U0, U, H, and G look absurd. As a comparison, look at MAE for mean value of each property:
These are in [eV] units (1 Ha = 27.212 eV). In other words, for instance MPNN models MAE for U property is 55.78 [eV] vs 8.25 [eV] for the mean baseline. This does not make sense at all. [k cal/mol] sounds more reasonable but then why half the energies in one unit the other half in other unit, and then why there are no units in a paper published by Royal Society of Chemistry. Seems like I'm missing something. Thanks!
quantum-chemistry machine-learning
asked 9 hours ago
BladeBlade
1306 bronze badges
$endgroup$
I am trying to compare the performance of few Quantum Chemistry property prediction ML models. I was looking at the following table from DOI:
10.1039/c7sc02664a
The problem is that it does not contain any units. A preprint (https://arxiv.org/pdf/1909.00259.pdf, Table 4) suggests that the energy units are Hartree. Among energy properties, HOMO, LUMO, and gap have reasonable values for Hartree unit. However, values reported for U0, U, H, and G look absurd. As a comparison, look at MAE for mean value of each property:
These are in [eV] units (1 Ha = 27.212 eV). In other words, for instance MPNN models MAE for U property is 55.78 [eV] vs 8.25 [eV] for the mean baseline. This does not make sense at all. [k cal/mol] sounds more reasonable but then why half the energies in one unit the other half in other unit, and then why there are no units in a paper published by Royal Society of Chemistry. Seems like I'm missing something. Thanks!
quantum-chemistry machine-learning
quantum-chemistry machine-learning
asked 9 hours ago
BladeBlade
1306 bronze badges
asked 9 hours ago
BladeBlade
1306 bronze badges
asked 9 hours ago
BladeBlade
1306 bronze badges
asked 9 hours ago
BladeBlade
1306 bronze badges
asked 9 hours ago
BladeBlade
1306 bronze badges
1306 bronze badges
add a comment
|
add a comment
|
1 Answer
1
active
oldest
votes
$begingroup$
Yes, you're exactly right - multiple papers in chemistry ML drop the units.
There are even comparisons (usually by statistics, ML or comp. sci. researchers) where models are compared by "averaging" errors down a column like that. Of course that's meaningless, since you can't average electron volts or Hartree (energies), Debye (dipole moments), and volume (polarizabilities).
Worse, energies are often computed as atomization energies - so for large molecules, they can be enormous...
In my opinion, a more relevant and meaningful statistic would be the Mean Absolute Percent Error (MAPE) which is unitless and easier to understand (e.g., 1% error? 0.01% error?)
That said, the original QM9 paper gives the units Table 3:
Scientific Data (2014) 1, art. 140022
answered 8 hours ago
Geoff HutchisonGeoff Hutchison
21.7k3 gold badges56 silver badges117 bronze badges
$endgroup$
1
$begingroup$
I'm adding an image of the data table - I believe this is under fair use, since one cannot copyright data - in this case the units of the QM9 set.
$endgroup$
– Geoff Hutchison
8 hours ago
$begingroup$
Thanks for the feedback @GeoffHutchison. I agree that it seems most reasonable to use datasets original units, but for atomization energies it doesn't seem to be right. I've decided to try recreate these results with deepchem. That seems to be the only way to find out.
$endgroup$
– Blade
6 hours ago
$begingroup$
Just to put it in perspective, the reason that I'm so suspicious is that if they are in fact using Ha units, then in a world that DFT error for U0 is 0.1 [eV] and people are already claiming to not only achieving this, but achieving chemical accuracy (0.04 [eV] I think), the Japanese paper is proudly reporting 1.35 [eV] and the benchmark method is 54.97 [eV]! So I believe that the Japanese paper is dead wrong (given that it's just on arxiv).
$endgroup$
– Blade
6 hours ago
1
$begingroup$
I have reviewed multiple manuscripts in the field that claim state-of-the-art accuracy when they're unaware of units or of other papers with better accuracy. I'm not sure what the "true" state is for U0 on QM9, but it's definitely < 1 kcal/mol
$endgroup$
– Geoff Hutchison
5 hours ago
add a comment
|
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "431"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/4.0/"u003ecc by-sa 4.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fchemistry.stackexchange.com%2fquestions%2f121744%2faccuracy-of-quantum-chemistry-ml-models%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
Yes, you're exactly right - multiple papers in chemistry ML drop the units.
There are even comparisons (usually by statistics, ML or comp. sci. researchers) where models are compared by "averaging" errors down a column like that. Of course that's meaningless, since you can't average electron volts or Hartree (energies), Debye (dipole moments), and volume (polarizabilities).
Worse, energies are often computed as atomization energies - so for large molecules, they can be enormous...
In my opinion, a more relevant and meaningful statistic would be the Mean Absolute Percent Error (MAPE) which is unitless and easier to understand (e.g., 1% error? 0.01% error?)
That said, the original QM9 paper gives the units Table 3:
Scientific Data (2014) 1, art. 140022
answered 8 hours ago
Geoff HutchisonGeoff Hutchison
21.7k3 gold badges56 silver badges117 bronze badges
$endgroup$
1
$begingroup$
I'm adding an image of the data table - I believe this is under fair use, since one cannot copyright data - in this case the units of the QM9 set.
$endgroup$
– Geoff Hutchison
8 hours ago
$begingroup$
Thanks for the feedback @GeoffHutchison. I agree that it seems most reasonable to use datasets original units, but for atomization energies it doesn't seem to be right. I've decided to try recreate these results with deepchem. That seems to be the only way to find out.
$endgroup$
– Blade
6 hours ago
$begingroup$
Just to put it in perspective, the reason that I'm so suspicious is that if they are in fact using Ha units, then in a world that DFT error for U0 is 0.1 [eV] and people are already claiming to not only achieving this, but achieving chemical accuracy (0.04 [eV] I think), the Japanese paper is proudly reporting 1.35 [eV] and the benchmark method is 54.97 [eV]! So I believe that the Japanese paper is dead wrong (given that it's just on arxiv).
$endgroup$
– Blade
6 hours ago
1
$begingroup$
I have reviewed multiple manuscripts in the field that claim state-of-the-art accuracy when they're unaware of units or of other papers with better accuracy. I'm not sure what the "true" state is for U0 on QM9, but it's definitely < 1 kcal/mol
$endgroup$
– Geoff Hutchison
5 hours ago
add a comment
|
$begingroup$
Yes, you're exactly right - multiple papers in chemistry ML drop the units.
There are even comparisons (usually by statistics, ML or comp. sci. researchers) where models are compared by "averaging" errors down a column like that. Of course that's meaningless, since you can't average electron volts or Hartree (energies), Debye (dipole moments), and volume (polarizabilities).
Worse, energies are often computed as atomization energies - so for large molecules, they can be enormous...
In my opinion, a more relevant and meaningful statistic would be the Mean Absolute Percent Error (MAPE) which is unitless and easier to understand (e.g., 1% error? 0.01% error?)
That said, the original QM9 paper gives the units Table 3:
Scientific Data (2014) 1, art. 140022
answered 8 hours ago
Geoff HutchisonGeoff Hutchison
21.7k3 gold badges56 silver badges117 bronze badges
$endgroup$
1
$begingroup$
I'm adding an image of the data table - I believe this is under fair use, since one cannot copyright data - in this case the units of the QM9 set.
$endgroup$
– Geoff Hutchison
8 hours ago
$begingroup$
Thanks for the feedback @GeoffHutchison. I agree that it seems most reasonable to use datasets original units, but for atomization energies it doesn't seem to be right. I've decided to try recreate these results with deepchem. That seems to be the only way to find out.
$endgroup$
– Blade
6 hours ago
$begingroup$
Just to put it in perspective, the reason that I'm so suspicious is that if they are in fact using Ha units, then in a world that DFT error for U0 is 0.1 [eV] and people are already claiming to not only achieving this, but achieving chemical accuracy (0.04 [eV] I think), the Japanese paper is proudly reporting 1.35 [eV] and the benchmark method is 54.97 [eV]! So I believe that the Japanese paper is dead wrong (given that it's just on arxiv).
$endgroup$
– Blade
6 hours ago
1
$begingroup$
I have reviewed multiple manuscripts in the field that claim state-of-the-art accuracy when they're unaware of units or of other papers with better accuracy. I'm not sure what the "true" state is for U0 on QM9, but it's definitely < 1 kcal/mol
$endgroup$
– Geoff Hutchison
5 hours ago
add a comment
|
$begingroup$
Yes, you're exactly right - multiple papers in chemistry ML drop the units.
There are even comparisons (usually by statistics, ML or comp. sci. researchers) where models are compared by "averaging" errors down a column like that. Of course that's meaningless, since you can't average electron volts or Hartree (energies), Debye (dipole moments), and volume (polarizabilities).
Worse, energies are often computed as atomization energies - so for large molecules, they can be enormous...
In my opinion, a more relevant and meaningful statistic would be the Mean Absolute Percent Error (MAPE) which is unitless and easier to understand (e.g., 1% error? 0.01% error?)
That said, the original QM9 paper gives the units Table 3:
Scientific Data (2014) 1, art. 140022
answered 8 hours ago
Geoff HutchisonGeoff Hutchison
21.7k3 gold badges56 silver badges117 bronze badges
$endgroup$
Yes, you're exactly right - multiple papers in chemistry ML drop the units.
There are even comparisons (usually by statistics, ML or comp. sci. researchers) where models are compared by "averaging" errors down a column like that. Of course that's meaningless, since you can't average electron volts or Hartree (energies), Debye (dipole moments), and volume (polarizabilities).
Worse, energies are often computed as atomization energies - so for large molecules, they can be enormous...
In my opinion, a more relevant and meaningful statistic would be the Mean Absolute Percent Error (MAPE) which is unitless and easier to understand (e.g., 1% error? 0.01% error?)
That said, the original QM9 paper gives the units Table 3:
Scientific Data (2014) 1, art. 140022
answered 8 hours ago
Geoff HutchisonGeoff Hutchison
21.7k3 gold badges56 silver badges117 bronze badges
answered 8 hours ago
Geoff HutchisonGeoff Hutchison
21.7k3 gold badges56 silver badges117 bronze badges
answered 8 hours ago
Geoff HutchisonGeoff Hutchison
21.7k3 gold badges56 silver badges117 bronze badges
answered 8 hours ago
Geoff HutchisonGeoff Hutchison
21.7k3 gold badges56 silver badges117 bronze badges
21.7k3 gold badges56 silver badges117 bronze badges
1
$begingroup$
I'm adding an image of the data table - I believe this is under fair use, since one cannot copyright data - in this case the units of the QM9 set.
$endgroup$
– Geoff Hutchison
8 hours ago
$begingroup$
Thanks for the feedback @GeoffHutchison. I agree that it seems most reasonable to use datasets original units, but for atomization energies it doesn't seem to be right. I've decided to try recreate these results with deepchem. That seems to be the only way to find out.
$endgroup$
– Blade
6 hours ago
$begingroup$
Just to put it in perspective, the reason that I'm so suspicious is that if they are in fact using Ha units, then in a world that DFT error for U0 is 0.1 [eV] and people are already claiming to not only achieving this, but achieving chemical accuracy (0.04 [eV] I think), the Japanese paper is proudly reporting 1.35 [eV] and the benchmark method is 54.97 [eV]! So I believe that the Japanese paper is dead wrong (given that it's just on arxiv).
$endgroup$
– Blade
6 hours ago
1
$begingroup$
I have reviewed multiple manuscripts in the field that claim state-of-the-art accuracy when they're unaware of units or of other papers with better accuracy. I'm not sure what the "true" state is for U0 on QM9, but it's definitely < 1 kcal/mol
$endgroup$
– Geoff Hutchison
5 hours ago
add a comment
|
1
$begingroup$
I'm adding an image of the data table - I believe this is under fair use, since one cannot copyright data - in this case the units of the QM9 set.
$endgroup$
– Geoff Hutchison
8 hours ago
$begingroup$
Thanks for the feedback @GeoffHutchison. I agree that it seems most reasonable to use datasets original units, but for atomization energies it doesn't seem to be right. I've decided to try recreate these results with deepchem. That seems to be the only way to find out.
$endgroup$
– Blade
6 hours ago
$begingroup$
Just to put it in perspective, the reason that I'm so suspicious is that if they are in fact using Ha units, then in a world that DFT error for U0 is 0.1 [eV] and people are already claiming to not only achieving this, but achieving chemical accuracy (0.04 [eV] I think), the Japanese paper is proudly reporting 1.35 [eV] and the benchmark method is 54.97 [eV]! So I believe that the Japanese paper is dead wrong (given that it's just on arxiv).
$endgroup$
– Blade
6 hours ago
1
$begingroup$
I have reviewed multiple manuscripts in the field that claim state-of-the-art accuracy when they're unaware of units or of other papers with better accuracy. I'm not sure what the "true" state is for U0 on QM9, but it's definitely < 1 kcal/mol
$endgroup$
– Geoff Hutchison
5 hours ago
1
1
$begingroup$
I'm adding an image of the data table - I believe this is under fair use, since one cannot copyright data - in this case the units of the QM9 set.
$endgroup$
– Geoff Hutchison
8 hours ago
$begingroup$
I'm adding an image of the data table - I believe this is under fair use, since one cannot copyright data - in this case the units of the QM9 set.
$endgroup$
– Geoff Hutchison
8 hours ago
$begingroup$
Thanks for the feedback @GeoffHutchison. I agree that it seems most reasonable to use datasets original units, but for atomization energies it doesn't seem to be right. I've decided to try recreate these results with deepchem. That seems to be the only way to find out.
$endgroup$
– Blade
6 hours ago
$begingroup$
Thanks for the feedback @GeoffHutchison. I agree that it seems most reasonable to use datasets original units, but for atomization energies it doesn't seem to be right. I've decided to try recreate these results with deepchem. That seems to be the only way to find out.
$endgroup$
– Blade
6 hours ago
$begingroup$
Just to put it in perspective, the reason that I'm so suspicious is that if they are in fact using Ha units, then in a world that DFT error for U0 is 0.1 [eV] and people are already claiming to not only achieving this, but achieving chemical accuracy (0.04 [eV] I think), the Japanese paper is proudly reporting 1.35 [eV] and the benchmark method is 54.97 [eV]! So I believe that the Japanese paper is dead wrong (given that it's just on arxiv).
$endgroup$
– Blade
6 hours ago
$begingroup$
Just to put it in perspective, the reason that I'm so suspicious is that if they are in fact using Ha units, then in a world that DFT error for U0 is 0.1 [eV] and people are already claiming to not only achieving this, but achieving chemical accuracy (0.04 [eV] I think), the Japanese paper is proudly reporting 1.35 [eV] and the benchmark method is 54.97 [eV]! So I believe that the Japanese paper is dead wrong (given that it's just on arxiv).
$endgroup$
– Blade
6 hours ago
1
1
$begingroup$
I have reviewed multiple manuscripts in the field that claim state-of-the-art accuracy when they're unaware of units or of other papers with better accuracy. I'm not sure what the "true" state is for U0 on QM9, but it's definitely < 1 kcal/mol
$endgroup$
– Geoff Hutchison
5 hours ago
$begingroup$
I have reviewed multiple manuscripts in the field that claim state-of-the-art accuracy when they're unaware of units or of other papers with better accuracy. I'm not sure what the "true" state is for U0 on QM9, but it's definitely < 1 kcal/mol
$endgroup$
– Geoff Hutchison
5 hours ago
add a comment
|
Thanks for contributing an answer to Chemistry Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fchemistry.stackexchange.com%2fquestions%2f121744%2faccuracy-of-quantum-chemistry-ml-models%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
