I co-authored a letter to the National AI Research Resource (NAIRR) Task Force, arguing that the NAIRR should “develop and implement a system for ‘compute accounting,’ standardized methods to track and audit the use of computational resources.”
Technology companies today already build tools to internally track compute resource usage. For example, in algorithmic stock trading, it is not uncommon to maintain a company-wide dashboard displaying how much compute is being used by which algorithms and for what purposes. Cloud computing platforms such as Google Cloud and Amazon Web Services offer similar services, but exact methods vary from company to company, and there is currently no industry-wide standard for compute accounting.
If the NAIRR management agency knows how much compute was expended to train each ML model, it can estimate the risk of misuse and assess whether compute is equitably distributed. Larger models, which require more computational resources, generally carry higher risks. The NAIRR management agency could conduct internal audits of the largest (highest risk) projects, to ensure those allocated resources are not being misused. Furthermore, paired with demographic information on NAIRR users, compute accounting data could be used to calculate the share of NAIRR resources that is supporting researchers from traditionally underserved communities.