For advanced DKIM setup this is usually done at the MTA such as postfix eg using a 'milter' like opendkim.
However there are advantages to doing this in Moodle directly such as when you have limited control over the way your email is being sent. Also by having it in Moodle it can be easier to manage.
In 3.10 / 4.0 a new setting was added that makes it possible to sign emails at the Moodle level and it requires setting up the private certificates and putting them in a known location where Moodle can find them. Because emails could be sent from a variety of From email addresses the location contains the domain in it's path and you can provide as many certificates as needed but this is an uncommon use case.
The most common setup
The simplest and fairly typical setup is where all emails are sent from the noreply email. In this setup we will give instructions on a linux setup such as debian or ubuntu.
$CFG->noreplyaddress = 'firstname.lastname@example.org'
In this case you need to choose a DKIM selector which is arbitrary but is often based on a date as the best practice is to rotate them on a periodic basis.
Lets say we have chosen a selector of '2020sep'.
Now in sitedata we need to create a folder to hold the DKIM certificate with a subdirectory matching the domain:
mkdir -p /path/to/sitedata/dkim/moodle.myschool.edu.au
Next in this directory generate the private key and public key DNS record using the opendkim-genkey tool:
opendkim-genkey -b 2048 -r -s 2020sep -d moodle.myschool.edu.au -v
This should result in two files like this:
Only the .private file is used by Moodle, the .txt file is the TXT record which you need to add to your DNS. To confirm that it is all correct there is a great public tool where you can enter the domain and DKIM selector and it will confirm the record looks like it is in the correct shape.
Once this is in place then use the email testing tool in moodle to send a test email, it can be useful to turn on the debugsmtp setting.
You should see the DKIM signature in the email headers. The email server receiving the email should also have validated this signature as well and added another header with the results of this validation.
ie in Gmail open the email, click the '...' on the right, then 'Show original' and in the headers it should say:
DKIM: 'PASS' with domain moodle.myschool.edu.au