{"id":2360,"date":"2021-06-19T13:00:00","date_gmt":"2021-06-19T12:00:00","guid":{"rendered":"https:\/\/chewett.co.uk\/blog\/?p=2360"},"modified":"2021-06-18T20:06:11","modified_gmt":"2021-06-18T19:06:11","slug":"learning-points-from-writing-my-own-js-css-caching-system","status":"publish","type":"post","link":"https:\/\/chewett.co.uk\/blog\/2360\/learning-points-from-writing-my-own-js-css-caching-system\/","title":{"rendered":"Learning points from writing my own JS\/CSS caching system"},"content":{"rendered":"\n<figure class=\"wp-block-image size-large\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" width=\"678\" height=\"254\" data-attachment-id=\"2945\" data-permalink=\"https:\/\/chewett.co.uk\/blog\/2360\/learning-points-from-writing-my-own-js-css-caching-system\/caching_system_header\/\" data-orig-file=\"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2021\/06\/caching_system_header.jpg?fit=800%2C300&amp;ssl=1\" data-orig-size=\"800,300\" data-comments-opened=\"1\" data-image-meta=\"{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}\" data-image-title=\"caching_system_header\" data-image-description=\"\" data-image-caption=\"\" data-medium-file=\"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2021\/06\/caching_system_header.jpg?fit=300%2C113&amp;ssl=1\" data-large-file=\"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2021\/06\/caching_system_header.jpg?fit=678%2C254&amp;ssl=1\" src=\"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2021\/06\/caching_system_header.jpg?resize=678%2C254&#038;ssl=1\" alt=\"\" class=\"wp-image-2945\" srcset=\"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2021\/06\/caching_system_header.jpg?w=800&amp;ssl=1 800w, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2021\/06\/caching_system_header.jpg?resize=300%2C113&amp;ssl=1 300w, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2021\/06\/caching_system_header.jpg?resize=768%2C288&amp;ssl=1 768w, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2021\/06\/caching_system_header.jpg?resize=50%2C19&amp;ssl=1 50w\" sizes=\"auto, (max-width: 678px) 100vw, 678px\" \/><\/figure>\n\n\n\n<p>This post talks about the various points I have learned from writing my own caching system, <a rel=\"noreferrer noopener\" aria-label=\"Cache-N-Crunch (opens in a new tab)\" href=\"https:\/\/github.com\/chewett\/cache-n-crunch\" target=\"_blank\">Cache-N-Crunch<\/a>.<\/p>\n\n\n\n<!--more-->\n\n\n\n<h2 class=\"wp-block-heading\">Why write your own caching system?<\/h2>\n\n\n\n<p>A while ago I wanted to minify the JavaScript and CSS on a website I worked on. The obvious solution would be to use one of the many caching libraries.<\/p>\n\n\n\n<p>I wanted to take advantage of merging all the files together and delivering a minified asset set.<\/p>\n\n\n\n<p>However there was one problem, the second developer didn&#8217;t want to use anything that wouldn&#8217;t automatically flush the cache. They typically devved directly on the main server as it was their testing server. This meant they required it to be simple to update as they were testing.<\/p>\n\n\n\n<p>Many of the caching layers I looked at require a cache clear. This was a manual step which created production ready assets. This wouldn&#8217;t be acceptable to him.<\/p>\n\n\n\n<p>Many of them were able to turn on development mode. In this mode it would continually regenerate the assets to use during development. But again running a command so they could work was not acceptable.<\/p>\n\n\n\n<p>So eventually I decided that I would learn the intricacies of caching and write one myself.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Compressing the assets<\/h2>\n\n\n\n<p>I wrote two small PHP wrappers around UglifyCSS and UglifyJS which allowed my caching library to minify the data. This simplified the logic to purely determining whether the caching needed to be invalidated and regenerated.<\/p>\n\n\n\n<p>This also benefitted from a pretty thorough compression program that was already used by many thousands of people.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Fulfilling the requirement of automatic compression<\/h2>\n\n\n\n<p>The primary requirement was that if a file has changed, it needs to be automatically compressed and minified for production. It also needs to be immediately reflected when devving. To achieve this I had two sets of logic, one for development and one for production use.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Logic for production mode<\/h3>\n\n\n\n<ul class=\"wp-block-list\"><li>If a minified asset is available serve that<\/li><li>If no minified asset is available, serve all assets that make up the minified asset directly.<\/li><\/ul>\n\n\n\n<p>This logic means that by default we will serve the minified assets. However if it isn&#8217;t available then it will serve the full set of assets that go into the minified final asset.<\/p>\n\n\n\n<p>No check for whether the file has changed would be performed in production mode. If there was a minified file it would be assumed to be the latest version. This sped up the production caching as it was a simple O(1) lookup to see if a minified asset was available. This assumes that any minified asset will be the latest version (which is a problem resolved by the dev mode logic).<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Logic for dev mode<\/h3>\n\n\n\n<ul class=\"wp-block-list\"><li>When loading an asset, if one of its components have changed, minify the assets<\/li><li>Once the asset has been fully minified, link to all component assets directly.<\/li><\/ul>\n\n\n\n<p>In dev mode, every time the page is loaded the assets are checked and minified if they have changed. This means also that the production assets are kept up to date as they are being worked on.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Determining if files have changed<\/h2>\n\n\n\n<p>To determine if the files have changed I would keep track of the hash of every file used when minified and then compare that when running in development mode.<\/p>\n\n\n\n<p>This kept the checks simple and relatively fast as running a simple hashing function on the files was quick. This did however  mean that changes to formatting that would have no effect on the code would cause it to be regenerated.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Side effects of merging files<\/h2>\n\n\n\n<p>Currently the files are merged together and minified as a group however this has some side effects. The main one I noticed was that since each file is merged, &#8220;use strict&#8221; caused issues.<\/p>\n\n\n\n<p>While some files were intended to run in strict mode, not all worked when set into this mode. Since the files were merged together, it caused all the  files merged after a strict mode file, to be ran in strict mode.<\/p>\n\n\n\n<p>This actually caused me to rework a number of the files running in strict mode. Amusingly this problem has also been found to happen on major websites too. Blindly merging files together therefore has some concerns depending how they are written!<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Smarter optimisation operations<\/h2>\n\n\n\n<p>While minifying does somewhat reduce the size while researching I found something more interesting.<\/p>\n\n\n\n<p>Webpack performs a technique called <a rel=\"noreferrer noopener\" aria-label=\"tree shaking (opens in a new tab)\" href=\"https:\/\/webpack.js.org\/guides\/tree-shaking\/\" target=\"_blank\">tree shaking<\/a> on the source. Here the source is statically analysed to work out what functions and variables are needed by each asset. During the minification process all unnecessary elements are removed from the final source.<\/p>\n\n\n\n<p>This is something that has significant advantages when used with ES2015 modules and is something my caching system did not implement. This could have been an improvement point however the code was not ES2015 compatible so likely would not have brought any benefit to that website.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">A conclusion on what I have learnt<\/h2>\n\n\n\n<p>Caching is a hard problem and you will run into unexpected problems when trying to roll your own. There are people much smarter than I who have written much more useful systems so always use a pre-built one if you can!<\/p>\n\n\n\n<p>If you do want to write your own its great fun, although can be a little mentally taxing! Lots of testing of different files helped when writing it. This alongside a codebase with a lot of different Javascript files helped me test it.<\/p>\n\n\n\n<p>This was certainly something that I enjoy doing but I still prefer using a pre-built minification system like webpack.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>This post talks about the various points I have learned from writing my own caching system, Cache-N-Crunch.<\/p>\n","protected":false},"author":1,"featured_media":2946,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"Today I am talking about Learning points from writing my own #JS\/#CSS caching system","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[98],"tags":[411,72,66,412],"class_list":["post-2360","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-software","tag-caching","tag-javascript","tag-php","tag-webpack"],"wppr_data":{"cwp_meta_box_check":"No"},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2021\/06\/caching_js_posticon_OUTPUT.png?fit=1200%2C628&ssl=1","jetpack_shortlink":"https:\/\/wp.me\/p2toWX-C4","jetpack_sharing_enabled":true,"jetpack-related-posts":[{"id":157,"url":"https:\/\/chewett.co.uk\/blog\/157\/selection-click-and-drag-doesnt-work-in-gmail-opera\/","url_meta":{"origin":2360,"position":0},"title":"Selection\/Click and drag doesnt work in Gmail &#8211; Opera","author":"Chewett","date":"September 27, 2013","format":false,"excerpt":"In opera normally click and drag or selection doesn't work. This is because in opera Gmail serves up some CSS that breaks the display and stops you clicking and dragging text. To fix this, you can enable opera's list of JavaScript changes, to make popular sites work. This is done\u2026","rel":"","context":"In &quot;Fixes&quot;","block_context":{"text":"Fixes","link":"https:\/\/chewett.co.uk\/blog\/category\/fixes\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1177,"url":"https:\/\/chewett.co.uk\/blog\/1177\/scaling-sprites-for-pixel-art-with-html5-canvas\/","url_meta":{"origin":2360,"position":1},"title":"Scaling Sprites for Pixel Art with HTML5 Canvas","author":"Chewett","date":"May 12, 2018","format":false,"excerpt":"In this blog post, I am showing a simple way of scaling sprites for Pixel Art\u00a0using the HTML5 canvas. Why we scale up sprites rather than rendering them higher quality When displaying pixel art it\u00a0is originally drawn pixel by pixel at a\u00a0very low resolution and then scaled up from original\u2026","rel":"","context":"In &quot;Software&quot;","block_context":{"text":"Software","link":"https:\/\/chewett.co.uk\/blog\/category\/software\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/05\/scaling_sprites-for_pixel_art.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/05\/scaling_sprites-for_pixel_art.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/05\/scaling_sprites-for_pixel_art.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/05\/scaling_sprites-for_pixel_art.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":1485,"url":"https:\/\/chewett.co.uk\/blog\/1485\/drawing-shapes-in-d3-js-version-5\/","url_meta":{"origin":2360,"position":2},"title":"Drawing shapes in D3.js Version 5","author":"Chewett","date":"September 5, 2018","format":false,"excerpt":"This post goes over the various D3.js symbols typically used for scatter plots. D3.js built-in symbols D3.js has a number of built-in symbols which can be used for any data visualisation needs. The most common use of these are\u00a0for points on scatter plots and similar graphs. It is important to\u2026","rel":"","context":"In &quot;Software&quot;","block_context":{"text":"Software","link":"https:\/\/chewett.co.uk\/blog\/category\/software\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/08\/d3_v5_drawing_shapes.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/08\/d3_v5_drawing_shapes.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/08\/d3_v5_drawing_shapes.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/08\/d3_v5_drawing_shapes.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":1184,"url":"https:\/\/chewett.co.uk\/blog\/1184\/copying-text-to-the-clipboard-with-javascript\/","url_meta":{"origin":2360,"position":3},"title":"Copying text to the clipboard with JavaScript","author":"Chewett","date":"May 26, 2018","format":false,"excerpt":"This post describes how you can copy text to your clipboard with JavaScript. Uses for copying text to the clipboard If you know there is something your user will want to copy giving them an easy way can to do this can be very helpful. This is even more important\u2026","rel":"","context":"In &quot;Software&quot;","block_context":{"text":"Software","link":"https:\/\/chewett.co.uk\/blog\/category\/software\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/05\/copying_text_to_clipboard.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/05\/copying_text_to_clipboard.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/05\/copying_text_to_clipboard.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/05\/copying_text_to_clipboard.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":1030,"url":"https:\/\/chewett.co.uk\/blog\/1030\/overlaying-geo-data-leaflet-version-1-3-d3-js-version-4\/","url_meta":{"origin":2360,"position":4},"title":"Overlaying geo data with Leaflet Version 1.3 and D3.js Version 4","author":"Chewett","date":"February 24, 2018","format":false,"excerpt":"In this post I describe how you can overlay Geo Data onto a leaflet map with D3.js. Combining Leaflet and D3 and objectives There are a number of tutorials online on how to overlay Geodata with D3.js. However, I couldn't find any of these that worked for the most recent\u2026","rel":"","context":"In &quot;Software&quot;","block_context":{"text":"Software","link":"https:\/\/chewett.co.uk\/blog\/category\/software\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/02\/d3_leaftlet_intro.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/02\/d3_leaftlet_intro.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/02\/d3_leaftlet_intro.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2018\/02\/d3_leaftlet_intro.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":2344,"url":"https:\/\/chewett.co.uk\/blog\/2344\/my-github-noops-hexbot\/","url_meta":{"origin":2360,"position":5},"title":"My Github Noops Hexbot","author":"Chewett","date":"September 7, 2019","format":false,"excerpt":"This blog post talks about my entry to the Github Noops Challenge for the Hexbot API. What the API does? This API lets you retrieve one or many random hex codes. This relatively simple API gives a lot of scope to try various different ways of using a hex code.\u2026","rel":"","context":"In &quot;Software&quot;","block_context":{"text":"Software","link":"https:\/\/chewett.co.uk\/blog\/category\/software\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/09\/github_hexbot.jpg?fit=800%2C800&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/09\/github_hexbot.jpg?fit=800%2C800&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/09\/github_hexbot.jpg?fit=800%2C800&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/chewett.co.uk\/blog\/wp-content\/uploads\/2019\/09\/github_hexbot.jpg?fit=800%2C800&ssl=1&resize=700%2C400 2x"},"classes":[]}],"_links":{"self":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts\/2360","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/comments?post=2360"}],"version-history":[{"count":7,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts\/2360\/revisions"}],"predecessor-version":[{"id":2948,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/posts\/2360\/revisions\/2948"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/media\/2946"}],"wp:attachment":[{"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/media?parent=2360"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/categories?post=2360"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/chewett.co.uk\/blog\/wp-json\/wp\/v2\/tags?post=2360"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}