Cache API - Revision history

Devdocsbot: Protected "Cache API": Developer Docs Migration ([Edit=Allow only administrators] (indefinite))

2023-12-01T08:28:10Z

Protected "Cache API": Developer Docs Migration ([Edit=Allow only administrators] (indefinite))

← Older revision		Revision as of 08:28, 1 December 2023
(No difference)

Devdocsbot: Update migration status and path

2023-12-01T08:25:17Z

Update migration status and path

Show changes

Brendanheywood: /* Setting requirements */

2023-05-10T13:29:45Z

Setting requirements

← Older revision		Revision as of 13:29, 10 May 2023
Line 142:		Line 142:
	; requiremultipleidentifiers : [bool] If set to true then only stores that support multiple identifiers will be used.		; requiremultipleidentifiers : [bool] If set to true then only stores that support multiple identifiers will be used.
	; requirelockingread : [bool] If set to true then a lock will be gained before reading from the cache store. It is recommended not to use this setting unless 100% absolutely positively required.		; requirelockingread : [bool] If set to true then a lock will be gained before reading from the cache store. It is recommended not to use this setting unless 100% absolutely positively required.
	; requirelockingbeforewrite : [bool] If set to true then ~~it expects the calling code to have gained~~ a lock ~~before attempting to write. The lock is~~ be held during expensive computation such as the generation of modinfo
	~~<br />Remember 99.9% of caches will NOT need this setting.<br />This setting will only be used for application caches presently.~~		; requirelockingbeforewrite :[bool] If set to true then a lock must be gained and held during expensive computation such as the generation of modinfo before writing to the cache store by the calling code. This is to prevent cache stampedes. After gaining the lock code must check to ensure the cache hasn't already been updated by another process. This is so far only used by course modinfo application caches presently.
	~~; requirelockingbeforewrite : [bool] If set to true then a lock must be gained~~ before writing to the cache store by the calling code. This is to prevent cache stampedes. After gaining the lock code must check to ensure the cache hasn't already been updated by another process. This is so far only used by course modinfo application caches presently.
	===Cache modifiers===		===Cache modifiers===
	You are also to modify the way in which the cache is going to operate when working for your definition.<br />		You are also to modify the way in which the cache is going to operate when working for your definition.<br />

Brendanheywood: Add docs for requirelockingbeforewrite

2023-05-10T13:28:14Z

Add docs for requirelockingbeforewrite

← Older revision		Revision as of 13:28, 10 May 2023
Line 115:		Line 115:
	'requirelockingread' => false,		'requirelockingread' => false,
	'requirelockingwrite' => false,		'requirelockingwrite' => false,
			'requirelockingbeforewrite' => false,
	'maxsize' => null,		'maxsize' => null,
	'overrideclass' => null,		'overrideclass' => null,
Line 140:		Line 141:
	; requiredataguarantee : [bool] If set to true then only stores that can guarantee data will remain available once set will be used.		; requiredataguarantee : [bool] If set to true then only stores that can guarantee data will remain available once set will be used.
	; requiremultipleidentifiers : [bool] If set to true then only stores that support multiple identifiers will be used.		; requiremultipleidentifiers : [bool] If set to true then only stores that support multiple identifiers will be used.
	; requirelockingread : [bool] If set to true then a lock will be gained before reading from the cache store. It is recommended not to use this setting unless 100% absolutely positively required.<br />Remember 99.9% of caches will NOT need this setting.<br />This setting will only be used for application caches presently.		; requirelockingread : [bool] If set to true then a lock will be gained before reading from the cache store. It is recommended not to use this setting unless 100% absolutely positively required.
	; ~~requirelockingwrite~~ : [bool] If set to true then a lock ~~will~~ be gained before writing to the cache store. ~~As above this~~ is ~~not recommended unless truly needed~~. ~~Please think about~~ the ~~order of your~~ code ~~and deal with race conditions there first~~.~~<br />~~This ~~setting will~~ only be used ~~for~~ application caches presently.		; requirelockingbeforewrite : [bool] If set to true then it expects the calling code to have gained a lock before attempting to write. The lock is be held during expensive computation such as the generation of modinfo
			<br />Remember 99.9% of caches will NOT need this setting.<br />This setting will only be used for application caches presently.
			; requirelockingbeforewrite : [bool] If set to true then a lock must be gained before writing to the cache store by the calling code. This is to prevent cache stampedes. After gaining the lock code must check to ensure the cache hasn't already been updated by another process. This is so far only used by course modinfo application caches presently.
	===Cache modifiers===		===Cache modifiers===
	You are also to modify the way in which the cache is going to operate when working for your definition.<br />		You are also to modify the way in which the cache is going to operate when working for your definition.<br />

Emerrill: MDL-42012 has been resolved in 3.8.6 and 3.9.3.

2022-06-11T03:42:30Z

MDL-42012 has been resolved in 3.8.6 and 3.9.3.

← Older revision		Revision as of 03:42, 11 June 2022
Line 169:		Line 169:
	; datasource : [string] A class to use as the data loader for this definition.<br />Any class used here must inherit the cache_data_source interface.		; datasource : [string] A class to use as the data loader for this definition.<br />Any class used here must inherit the cache_data_source interface.
	; datasourcefile : [string] Suplements the above setting indicated the file containing the class to be used. This file is included when required.		; datasourcefile : [string] Suplements the above setting indicated the file containing the class to be used. This file is included when required.
	GOTCHA: Note that if caching is disabled then nothing will be loaded through the data source which is probably not what you expect (rather than the data source being loaded every time but never cached). See also:		GOTCHA: Note that, in Moodle versions prior to 3.8.6 and 3.9.3, if caching is disabled then nothing will be loaded through the data source which is probably not what you expect (rather than the data source being loaded every time but never cached). See also:

	https://tracker.moodle.org/browse/MDL-42012		https://tracker.moodle.org/browse/MDL-42012

Brendanheywood: /* Misc settings */

2021-10-26T02:08:11Z

Misc settings

Brendanheywood: Added section of value versioning vs key versioning

2021-10-21T11:30:27Z

Added section of value versioning vs key versioning

← Older revision		Revision as of 11:30, 21 October 2021
Line 200:		Line 200:

	https://github.com/moodle/moodle/blob/master/lib/outputlib.php#L88-L100		https://github.com/moodle/moodle/blob/master/lib/outputlib.php#L88-L100

			It works best with a cache store that supports Least Recently Used garbage collection.

			=== Revision numbers as value suffix instead of key suffix ===
			This is conceptually the same as above but has two important differences. Firstly because the key itself is always the same then only a single version of some value will be stored on disk or in memory for any given cache store which makes it much more efficient in terms of storage. But secondly this comes with a higher coding complexity cost because it will no longer be guaranteed to be correct because a local cache could return a hit with a stale version. So if you need it to be correct you will need to parse out the revision from the value and then confirm the revision is correct before using the value. If it is invalid then you need to treat it as a miss and handle that. One way is to rebuild and set it, but this loses the advantage of primary and final caches (see below). A better way is to delete the local cache but not the final cache by passing a second param of false to delete, and then getting the value again which will repopulate the local cache from the shared cache:
			$cache->delete('foo', false); // Delete the primary / local stale version
			$cache->get('foo'); // Get the final / shared version (which may also be stale!)
			But this also is imperfect if there are 3 layers of caching, see [https://tracker.moodle.org/browse/MDL-72837 MDL-72837] for a full discussion and a possible new api to handle this.

			This method is how the course modinfo cache is localized.

	=== Using time as a key suffix ===		=== Using time as a key suffix ===
	Another common approach is to use a timestamp, this is how some of the core cache numbers work, see increment_revision_number() for some examples. This has the benefit of not needing any transaction or locking, but you do run the risk of two processes clashing if they happen to run in the same second.		Another common approach is to use a timestamp, this is how some of the core cache numbers work, see increment_revision_number() for some examples. This has the benefit of not needing any transaction or locking, but you do run the risk of two processes clashing if they happen to run in the same second.
Line 206:		Line 217:

	https://github.com/moodle/moodle/blob/master/lib/datalib.php#L1131-L1145		https://github.com/moodle/moodle/blob/master/lib/datalib.php#L1131-L1145

			It works best with a cache store that supports Least Recently Used garbage collection.
	=== Content hashes as keys ===		=== Content hashes as keys ===
	A great strategy is to use a digest or hash such as sha1 / sha256 of the content stored inside the cache, or in even more advanced scenarios a hash of the dependencies of the content in the cache. This guarantees that the key will be unique, and can be truly distributed without any synchronous communication between the front ends.		A great strategy is to use a digest or hash such as sha1 / sha256 of the content stored inside the cache, or in even more advanced scenarios a hash of the dependencies of the content in the cache. This guarantees that the key will be unique, and can be truly distributed without any synchronous communication between the front ends.
Line 216:		Line 229:

	This is a very common strategy is many distributed systems, outside of the context of caching, and is conceptually how git works internally which is why commits are a sha1 hash.		This is a very common strategy is many distributed systems, outside of the context of caching, and is conceptually how git works internally which is why commits are a sha1 hash.

			This strategy works well if you may periodically change back and forth to previous states which will still be present and so be immediate hits without re-warming. It works best with a cache store that supports Least Recently Used garbage collection.
	=== Primary and Final caches ===		=== Primary and Final caches ===
	Because http requests are generally assigned randomly or round-robin to front ends, when a cache item version changes you will now effectively have an empty cache on every front end. As you get the same cache item again and again on each front end it will continue to be a cache miss on each local box until they are all warm, which can ironically mean that on average for a fast changing, but not often requested cache item, your cache hit rate will be very low and much worse than if you had a shared cache. The solution here is to have multiple levels of caches setup, a local cache backed by a shared cache. You do not need to do anything special in the code to support this if your cache is already localizable, the MUC manages this for you, ie if you request a key which is missing from the local cache it will then request it from the shared cache. If present it will copy it back to the local cache and then return the value. If it is not present in either then your fall back code will generate the item, and it will be written to both cache stores at the same time.		Because http requests are generally assigned randomly or round-robin to front ends, when a cache item version changes you will now effectively have an empty cache on every front end. As you get the same cache item again and again on each front end it will continue to be a cache miss on each local box until they are all warm, which can ironically mean that on average for a fast changing, but not often requested cache item, your cache hit rate will be very low and much worse than if you had a shared cache. The solution here is to have multiple levels of caches setup, a local cache backed by a shared cache. You do not need to do anything special in the code to support this if your cache is already localizable, the MUC manages this for you, ie if you request a key which is missing from the local cache it will then request it from the shared cache. If present it will copy it back to the local cache and then return the value. If it is not present in either then your fall back code will generate the item, and it will be written to both cache stores at the same time.
Line 230:		Line 245:
	=== Time-To-Live for distributed caches ===		=== Time-To-Live for distributed caches ===
	Another consideration is the total size of your cache stores across all the front ends. As cache keys change they are never be invalidated or purged. So you should have in place some process to garbage collect stale items. This is more a concern for the cache store implementations and the configuration but worth considering. Some stores are deleted on upgrade, or have a Time-To-Live or a Least Recently Used strategy for deleting stale items.		Another consideration is the total size of your cache stores across all the front ends. As cache keys change they are never be invalidated or purged. So you should have in place some process to garbage collect stale items. This is more a concern for the cache store implementations and the configuration but worth considering. Some stores are deleted on upgrade, or have a Time-To-Live or a Least Recently Used strategy for deleting stale items.

	==Using a data source==		==Using a data source==

Brendanheywood: /* Using time as a key suffix */

2021-09-02T03:49:08Z

Using time as a key suffix

← Older revision		Revision as of 03:49, 2 September 2021
Line 201:		Line 201:
	https://github.com/moodle/moodle/blob/master/lib/outputlib.php#L88-L100		https://github.com/moodle/moodle/blob/master/lib/outputlib.php#L88-L100
	=== Using time as a key suffix ===		=== Using time as a key suffix ===
	Another common approach is to use a timestamp, this is how some of the core cache numbers work, see increment_revision_number() for some examples. This has the benefit of not needing any transaction or locking, but you do run the risk of two processes clashing if they happen to run in the same second~~, so generally don't do this~~.		Another common approach is to use a timestamp, this is how some of the core cache numbers work, see increment_revision_number() for some examples. This has the benefit of not needing any transaction or locking, but you do run the risk of two processes clashing if they happen to run in the same second.

	It may look like Moodle theme-related caching uses this ~~strategry~~, but actually if you look at [https://github.com/moodle/moodle/blob/df0e58adb140f90712bcd3229ad936d3b4bc15d9/lib/outputlib.php#L88 the code for theme_get_next_revision], you will see that it is actually guaranteeing to generate a unique new integer. It is just making that integer close to current time-stamp, to make it more self-documenting.		It may look like Moodle theme-related caching uses this strategy, but actually if you look at [https://github.com/moodle/moodle/blob/df0e58adb140f90712bcd3229ad936d3b4bc15d9/lib/outputlib.php#L88 the code for theme_get_next_revision], you will see that it is actually guaranteeing to generate a unique new integer which mitigates the clashing mentioned above. It is just making that integer close to current time-stamp, to make it more self-documenting.

	https://github.com/moodle/moodle/blob/master/lib/datalib.php#L1131-L1145		https://github.com/moodle/moodle/blob/master/lib/datalib.php#L1131-L1145
	=== Content hashes as keys ===		=== Content hashes as keys ===
	A great strategy is to use a hash such as sha1 / sha256 of the content stored inside the cache, or in even more advanced scenarios a hash of the dependencies of the content in the cache. This guarantees that the key will be unique, and can be truly distributed without any synchronous communication between the front ends.		A great strategy is to use a digest or hash such as sha1 / sha256 of the content stored inside the cache, or in even more advanced scenarios a hash of the dependencies of the content in the cache. This guarantees that the key will be unique, and can be truly distributed without any synchronous communication between the front ends.

	This strategy can work very well when building up large cache items from many smaller cache fragments in a nested tree structure, eg when caching a structure which is built of other cache items, eg 10 chunks of html to get combined into a large chunk to be cached, you would append the 10 hashes of the 10 smaller chunks and then hash that to use as the key for the combined cache item. This means you can mutate a small part of a tree structure and quickly re-generate the whole tree without expensively regenerating all of the other branches and leaves in the tree.		This strategy can work very well when building up large cache items from many smaller cache fragments in a nested tree structure, eg when caching a structure which is built of other cache items, eg 10 chunks of html to get combined into a large chunk to be cached, you would append the 10 hashes of the 10 smaller chunks and then hash that to use as the key for the combined cache item. This means you can mutate a small part of a tree structure and quickly re-generate the whole tree without expensively regenerating all of the other branches and leaves in the tree.
Line 215:		Line 215:
	https://blog.appsignal.com/2018/04/03/russian-doll-caching-in-rails.html		https://blog.appsignal.com/2018/04/03/russian-doll-caching-in-rails.html

	This is a very common strategy is many distributed systems, outside of the context of caching, and is conceptually how git works internally.		This is a very common strategy is many distributed systems, outside of the context of caching, and is conceptually how git works internally which is why commits are a sha1 hash.
	=== Primary and Final caches ===		=== Primary and Final caches ===
	Because http requests are generally assigned randomly or round-robin to front ends, when a cache item version changes you will now effectively have an empty cache on every front end. As you get the same cache item again and again on each front end it will continue to be a cache miss on each local box until they are all warm, which can ironically mean that on average for a fast changing, but not often requested cache item, your cache hit rate will be very low and much worse than if you had a shared cache. The solution here is to have multiple levels of caches setup, a local cache backed by a shared cache. You do not need to do anything special in the code to support this, the MUC manages this for you, ie if you request a key is missing from the local cache it will then request it from the shared cache. If present it will copy it back to the local cache. If it is not present in either then your fall back code will generate the item, and it will be written to both cache stores at the same time.		Because http requests are generally assigned randomly or round-robin to front ends, when a cache item version changes you will now effectively have an empty cache on every front end. As you get the same cache item again and again on each front end it will continue to be a cache miss on each local box until they are all warm, which can ironically mean that on average for a fast changing, but not often requested cache item, your cache hit rate will be very low and much worse than if you had a shared cache. The solution here is to have multiple levels of caches setup, a local cache backed by a shared cache. You do not need to do anything special in the code to support this if your cache is already localizable, the MUC manages this for you, ie if you request a key which is missing from the local cache it will then request it from the shared cache. If present it will copy it back to the local cache and then return the value. If it is not present in either then your fall back code will generate the item, and it will be written to both cache stores at the same time.

	This is especially important if you are scaling up and down the number of frontends quickly. If you suddenly need more horizontal scale and create a bunch of new front ends with empty caches and no shared cache they will all consume even more resources warming up and loading the shared services such as the database and filesystem.		This is especially important if you are scaling up and down the number of frontends quickly. If you suddenly need more horizontal scale and create a bunch of new front ends with empty caches and no shared cache they will all consume even more resources warming up and loading the shared services such as the database and filesystem.
Line 225:		Line 225:
	As you scale even bigger, a new bottleneck can appear when purging a shared disk cache ie when you deploy a new version. A full purge needs to iterate over and remove and sync a very large number of files, which can take some time. See MDL-69088 for a proposed fix.		As you scale even bigger, a new bottleneck can appear when purging a shared disk cache ie when you deploy a new version. A full purge needs to iterate over and remove and sync a very large number of files, which can take some time. See MDL-69088 for a proposed fix.
	=== Beware fast churning keys ===		=== Beware fast churning keys ===
	A big concern when designing a cache is how fast you anticipate it ~~to change~~. If it contains very fast moving data but sparsely requested data (see above) then you can end up in a situation where you are effectively just using the shared final cache, and wasting latency and space and IO cloning data to the local cache where is may not be hit again very much. As always caching is a balancing act trading off between CPU, time and disk, and ultimately money.		A big concern when designing a cache is how fast you anticipate it changing. If it contains very fast moving data but sparsely requested data (see above) then you can end up in a situation where you are effectively just using the shared final cache, and wasting latency and space and IO cloning data to the local cache where is may not be hit again very much. As always caching is a balancing act trading off between CPU, time and disk, and ultimately money.

	Even if you cache is able to use a local store that doesn't mean it actually will be configured to be local (and you can't tell either way). So a wasteful cache item will consume much more space storing all the previous versions of its items even if it isn't localized, and it will be much worse if it is.
	~~=== Time to life for distributed caches ===~~
	Another consideration is the total size of your cache stores across all the front ends. As cache keys change they are never invalidated or purged. So you should have in place some process to garbage collect stale items. This is more a concern for the cache store implementations and the configuration but worth considering. Some stores are deleted on upgrade, or have a Time-To-Live or a Least Recently Used strategy for deleting stale items.

			Even if your cache is able to use a local store that doesn't mean it actually will be configured to be local (and your code can't tell either way). So a wasteful cache item will consume much more space storing all the previous versions of its items even if it isn't localized, and it will be much worse if it is.
			=== Time-To-Live for distributed caches ===
			Another consideration is the total size of your cache stores across all the front ends. As cache keys change they are never be invalidated or purged. So you should have in place some process to garbage collect stale items. This is more a concern for the cache store implementations and the configuration but worth considering. Some stores are deleted on upgrade, or have a Time-To-Live or a Least Recently Used strategy for deleting stale items.

	==Using a data source==		==Using a data source==

Brendanheywood: /* Revision numbers as key suffix */

2021-09-02T03:40:17Z

Revision numbers as key suffix

← Older revision		Revision as of 03:40, 2 September 2021
Line 197:		Line 197:
	One potential benefit of a simple version number strategy if your cache misses are very expensive, is that you can check for the presence of a version, and if it doesn't exist it is easy to simply retrieve the previous version and use it in the mean time, while you could generate the new version asynchronously. This is an advanced concept and depends on the use case and has the obvious disadvantage of showing stale data.		One potential benefit of a simple version number strategy if your cache misses are very expensive, is that you can check for the presence of a version, and if it doesn't exist it is easy to simply retrieve the previous version and use it in the mean time, while you could generate the new version asynchronously. This is an advanced concept and depends on the use case and has the obvious disadvantage of showing stale data.

	~~And~~ example of this strategy in Moodle is the theme versions cache (~~not~~ this is not actually in MUC but the concepts are the same):		An example of this strategy in Moodle is the theme versions cache (NOTE: this is not actually in MUC but the caching concepts are the same):

	https://github.com/moodle/moodle/blob/master/lib/outputlib.php#L88-L100		https://github.com/moodle/moodle/blob/master/lib/outputlib.php#L88-L100

Brendanheywood: /* When should you localize? */

2021-08-13T06:45:54Z

When should you localize?

← Older revision		Revision as of 06:45, 13 August 2021
Line 189:		Line 189:
	=== When should you localize? ===		=== When should you localize? ===
	Not all caches are good candidates to localize and some can have a detrimental effect if localized for the wrong reasons.		Not all caches are good candidates to localize and some can have a detrimental effect if localized for the wrong reasons.
	[[File:When to localize cache.png\|alt=When to localize a cache\|none\|thumb\|~~700x700px~~\|When to localize a cache]]		[[File:When to localize cache.png\|alt=When to localize a cache\|none\|thumb\|683x683px\|When to localize a cache]]

	=== Revision numbers as key suffix ===		=== Revision numbers as key suffix ===
	A simple method is storing a version number somewhere and appending that to your key. This is the most common method used in Moodle, simply store a number somewhere which is globally shared such as in a custom db field, or using set_config / get_config.		A simple method is storing a version number somewhere and appending that to your key. This is the most common method used in Moodle, simply store a number somewhere which is globally shared such as in a custom db field, or using set_config / get_config.

← Older revision		Revision as of 02:08, 26 October 2021
Line 174:		Line 174:
	===Misc settings===		===Misc settings===
	The following are stand along settings that don't fall into any of the above categories.		The following are stand along settings that don't fall into any of the above categories.
	; invalidationevents : [array] An array of events that should cause this cache to invalidate some or all of the items within it.		; invalidationevents :[array] An array of events that should cause this cache to invalidate some or all of the items within it. Note that these are NOT normal moodle events and predates the [[Events API]] . Instead these are arbitrary strings which can be used by cache_helper::purge_by_event('changesincoursecat'); to mark multiple caches as invalid at once without the calling code knowing which caches are affected.
	; mappingsonly : [bool] If set to true only the mapped cache store(s) will be used and the default mode store will not. This is a super advanced setting and should not be used unless absolutely required. It allows you to avoid the default stores for one reason or another.		; mappingsonly : [bool] If set to true only the mapped cache store(s) will be used and the default mode store will not. This is a super advanced setting and should not be used unless absolutely required. It allows you to avoid the default stores for one reason or another.
	; sharingoptions : [int] The sharing options that are appropriate for this definition. Should be the sum of the possible options.		; sharingoptions : [int] The sharing options that are appropriate for this definition. Should be the sum of the possible options.
Line 202:		Line 202:

	It works best with a cache store that supports Least Recently Used garbage collection.		It works best with a cache store that supports Least Recently Used garbage collection.

	=== Revision numbers as value suffix instead of key suffix ===		=== Revision numbers as value suffix instead of key suffix ===
	This is conceptually the same as above but has two important differences. Firstly because the key itself is always the same then only a single version of some value will be stored on disk or in memory for any given cache store which makes it much more efficient in terms of storage. But secondly this comes with a higher coding complexity cost because it will no longer be guaranteed to be correct because a local cache could return a hit with a stale version. So if you need it to be correct you will need to parse out the revision from the value and then confirm the revision is correct before using the value. If it is invalid then you need to treat it as a miss and handle that. One way is to rebuild and set it, but this loses the advantage of primary and final caches (see below). A better way is to delete the local cache but not the final cache by passing a second param of false to delete, and then getting the value again which will repopulate the local cache from the shared cache:		This is conceptually the same as above but has two important differences. Firstly because the key itself is always the same then only a single version of some value will be stored on disk or in memory for any given cache store which makes it much more efficient in terms of storage. But secondly this comes with a higher coding complexity cost because it will no longer be guaranteed to be correct because a local cache could return a hit with a stale version. So if you need it to be correct you will need to parse out the revision from the value and then confirm the revision is correct before using the value. If it is invalid then you need to treat it as a miss and handle that. One way is to rebuild and set it, but this loses the advantage of primary and final caches (see below). A better way is to delete the local cache but not the final cache by passing a second param of false to delete, and then getting the value again which will repopulate the local cache from the shared cache:
Line 210:		Line 209:

	This method is how the course modinfo cache is localized.		This method is how the course modinfo cache is localized.

	=== Using time as a key suffix ===		=== Using time as a key suffix ===
	Another common approach is to use a timestamp, this is how some of the core cache numbers work, see increment_revision_number() for some examples. This has the benefit of not needing any transaction or locking, but you do run the risk of two processes clashing if they happen to run in the same second.		Another common approach is to use a timestamp, this is how some of the core cache numbers work, see increment_revision_number() for some examples. This has the benefit of not needing any transaction or locking, but you do run the risk of two processes clashing if they happen to run in the same second.