4

Fundamentals all the way down

 1 year ago
source link: https://changelog.com/jsparty/283
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
neoserver,ios ssh client

Transcript

šŸ“ Edit Transcript

Changelog

Click here to listen along while you enjoy the transcript. šŸŽ§

Hoy-hoy. Welcome to another exciting JS Party. Iā€™m your host this week, Nick Nisi, and I am joined by two wonderful regulars, and that is Kball - Kball, howā€™s it going?

Going alright, coming in hot.

[laughs] Yes, as always. And also Jerodā€™s here. Jerod, howā€™s it going?

Itā€™s going great, because Iā€™m here with you, Iā€™m here with Kball, and my ears have this ringing, and Iā€™m not sure why.

Thatā€™s just my voiceā€¦

I think it might have been your fault.

Itā€™s the sound of angels.

And that angelic voice that youā€™ve heard is Austin Gil joining us again. Austin, howā€™s it going?

Hey. Iā€™m doing well, living the good summer life here in Portland, taking a break from camping, and river floats, and bike rides to come hang out with you all. Itā€™s such a pleasure.

Good to have you.

River float sounds goodā€¦

Come by sometime. I will take you. I will make sure anyone on the show, anyone watching, anyone listening - come visit, I will show you a good time in Portland.

No promises there, but I can confirm that I will see you in person in Lincoln, Nebraska next month. So thatā€™s pretty exciting.

Thatā€™s right. Iā€™m very excited.

Do you do river floats there?

Noā€¦ Well, I think there is a riverā€¦

We have rivers. Come on, Nick, donā€™t sell us short.

We do have ā€“ do we have a river in Lincoln?

Oh, Lincoln. No. Nothing good ever came out of Lincoln. Just kiddingā€¦ Sort of.

[laughs] But very exciting. Weā€™re gonna meet up in person for the first time, and - I just got my ticket yesterday.

Where are you guys gonna meet up?

Lincoln, it sounds likeā€¦

Yeah, everybodyā€™s going to Lincoln.

But why? But why? Is there a reason?

I donā€™t know, Nebraska, but Iā€™m guessing thatā€™s like five people, or something. Itā€™s easy to find.

Well, yeah. Iā€™ll be there for Nebraska Codes. It should be a fun conference. Iā€™ll be speaking there, so if anyone wants to come find me - yeah, letā€™s be friends.

Now, Austin, we had you on - letā€™s seeā€¦ It was quite a while ago; I think it was episode 243, and we were talking about HTML access without JavaScript from your camera. And that was a fascinating article. So we were kind of talking about what youā€™ve been up to lately, and it sounds like a little more digging into the file system APIā€¦ Why donā€™t you catch us up on what youā€™ve been doing?

Yeah, so I donā€™t know why I get into these deep-dives into fundamentals ā€“ well, we use fundamentals as a loaded termā€¦ But yeah, into like fundamentals of whatā€™s available in the programming languages, and frameworks, and tools, and platforms that I useā€¦ So I did a deep-dive on forms, and then that led me into accessing files with just HTML, or accessing camera with just HTMLā€¦ And then recently, I have been on this sort of quest from God on trying to promote people to understand fundamentals and use fundamentals, as opposed to leaning too heavily on libraries, and frameworks, and tools like that.

And I was on a conversation recently with JavaScript Jabber guys, and we talked about just the fundamentals, and rehashed this familiar conversation of ā€œShould you focus on fundamentals, or should you focus on frameworks and libraries and tools?ā€ and it became a very nebulous conversationā€¦ So Iā€™d like to keep promoting peopleā€™s understanding of fundamentals, without getting too into like heady/space/galaxy brain stuffā€¦ You know, maybe get into some more practical information today.

Where does the Galaxy brain land with regards to fundamentals?

Well, I just think ā€“ I guess nebulous is the best term. Itā€™s a loaded term that doesnā€™t have a whole lot of clear definition of what youā€™re talking about when you mean the fundamentals. I think the best that Iā€™ve come to is saying that the fundamentals are sort of the lower-level knowledge that you need in order to explain the topic that youā€™re discussing about. So when weā€™re talking about fundamentals, I think a good framework to understand it is to say take a tool like Next.js - Next.js has a whole bunch of things that it makes available just within its sort of ecosystem, and building a Next.js application requires understanding those fundamental principles of that tool. But Next.js is built on top of React, and there are boundaries between what Next offers and what React offers. And React is like the lower-level primitive of that metaframework. And then React is built on top of JavaScript, so thereā€™s fundamental knowledge that you need to understand the separation of concerns between JavaScript and Reactā€¦ And then when youā€™re with JavaScript, then you have to understand the platform, andā€¦

Itā€™s turtles all the way down. We could take this down and say, ā€œOkay, now we need to understand interpreters, and interpreted language, and that means we need to understand machine code, and compilersā€, and like all these different layers.

But that is what Iā€™m saying. But you have to find where you live in the layers, right? Like, where are you right now, and the work that youā€™re doing - you need to understand the fundamentals of that. You donā€™t need to go all the way down to ones and zeros, to understand everything. But say, ā€œWhat is the code that Iā€™m writing doing, and what of that code lives within the current stack, or the current layer of the stack, and what belongs to the next layer down?ā€

And I think at some point you do have to get down to like lower-level things, like HTTP, the protocols, and the platforms. And then at that point ā€“

I mean, some of this comes to how leaky are your abstractions.

Exactly, yes. Yes.

[06:01] The web platform and JavaScript provide pretty solid abstractions. Itā€™s pretty rare that those leak and you have to go lower than that if youā€™re writing for the web. Next, or Nuxt, has some leaky spots.

Yeah, yeah. And thatā€™s kind of what Iā€™m promoting, is mostly the curiosity to take a moment and check your knowledge of understanding where those abstractions are, and who is responsible for those. And it just helps you write better software.

Please allow me to make this more abstract by introducing a metaphor, because I feel like this would be a great episode if we never let Austin get to his actual fundamental topic, and we just keep goingā€¦

[laughs] This is exactly how the last conversation wentā€¦ Itā€™s great.

So as a sports ball coach - you know, we coach basketball. And when you get the kids in the room, and you give them a ball, and you say ā€œOkay, itā€™s time for fundamentals.ā€ And Kball, we teach them dribbling, and passing, and shooting, and defense. Thatā€™s basically the fundamentals. Weā€™re not gonna teach them gravity, or how to tie their shoes, or stuff like this, which actually are lower-level than the fundamentals, but theyā€™re not the fundamentals of basketball.

Now, having said that, what they like to do though is not come in and learn how to dribble; what they like to do is come to half court and just chuck the ball at the rim.

Scrimmage!

Yeah, see if they can get a half court shot to go in. Thatā€™s what they want to do. And so thereā€™s your tie back to this desire to go straight to the end game, which is what we do a lot of times in web development. Itā€™s like ā€œWell, Iā€™m trying to get things done, and so Iā€™m gonna grab Next.js, and Iā€™m gonna maybe skip the dribbling and the passing and the shooting fundamentals. I can backfill those later.ā€ Austin is here to teach us how to dribble.

I think one of the key questions there is how you make dribbling fun, right? Some people - Iā€™m gonna run with your metaphor, except for me itā€™s coaching soccer. Iā€™ve been coaching soccer, rather than basketball.

Okay, fair enough.

But you still have dribbling. We have that in common.

So there are kids that they can map that ā€œIf I get better at this, Iā€™m going to be better at that.ā€ I think the same is true for adults. There are adults who can say, ā€œOkay, if I focus on this thing, thatā€™s going to eventually help me with that, so itā€™s worth doing that.ā€ And there are kids where if you have them out there doing drills, they check out. Theyā€™re not getting anything from that fundamentals work, because theyā€™re like ā€œI donā€™t see ā€“ whatā€™s the point? This is boring. Iā€™m gonna go chase butterflies.ā€ But if you get them in something that feels realā€¦ You know, a little scrimmage, maybe you narrow it down to like a 2-1-2, so they get lots of touches on the ball, they get lots of opportunity to start to move their body in these patternsā€¦ And you can give them little nudges here and there and say, ā€œOkay, there you want to kick it a little bit more like this. Or that was a good opportunity to do a little bit more dribblingā€, but itā€™s within a framework that looks a lot more like the real thing.

So Iā€™m going to pull this back then to Austin - where is your 2-1-2? Whereā€™s your 2-1-2 when it comes to web fundamentals?

Wow. Tie that back, Austin.

Uhhhā€¦ [laughter] I donā€™t know if I can answer that. Iā€™m just gonna deflect that and say that ā€“ I think that you touched on a really good point, that I wanted to bring back, which is being able to see your developmental path. And if you want to get better at software engineering, then I think understanding the fundamentals helps with that, because while youā€™re writing code, if you can periodically check in with yourself and say, ā€œHow much of this process or this functionality that Iā€™m building do I understand what a tool or library or framework is doing?ā€, to be able to explain and say, ā€œOkay, itā€™s definitely good that Iā€™m using this tool, because if I didnā€™t, Iā€™d know what it would take to rebuild the same functionality, and thereā€™s too much work. Like, that abstraction is worth keeping.ā€ And otherwise, you can say ā€œThat abstraction may not be worth keepingā€, because that can paint you into a corner. That can be limiting to working with the API that they provide, and not be able to take advantage of certain features that are at that lower level, that could be introducing more blow into client-side tooling, because youā€™re including a library that you really only need a small subset of the fundamentals, of what the platform can provide, that this library builds on top of, and introduces more work to download; libraries can also introduce more maintenance headache, can introduce more security vulnerabilities in addition to that bloat, can introduce that story where you come back to a project after six months and the upgrade story is a nightmare, because Node version compatibility is broken, or things like thatā€¦ Itā€™s just itā€™s kind of a nightmare.

[10:43] It doesnā€™t have to be that wayā€¦

It doesnā€™t have to be that wayā€¦

But it is that way.

But it kind of is that way. Yeah. Like, itā€™s been that way for my career.

[laughs] Letā€™s be real.

Maybe it doesnā€™t in the future, butā€¦ I donā€™t know, thereā€™s actually ā€“ well, thatā€™s a story for another time. But I think one of the most valuable skills that is underrated in terms of using tooling, or opting into using tooling, is as soon as you start relying on frameworks to do the work for you, you give up transferable knowledge. And when you work at a lower level - this is a JavaScript podcast. So when I have ā€“

Ostensibly.

Yeah. [laughs] When I have code thatā€™s like - letā€™s say a random string generator to generate IDs for me; if Iā€™m using that from a package that I rely on npm, or a tool thatā€™s available in like Solid.js, all of a sudden I have a dependency on that, and I canā€™t take that code. Itā€™s not portable to take to other projects that are built on top of different tooling, and itā€™s not knowledge that I can even transfer myself. So I like to have a snippet library in my GitHub thatā€™s like all of these little low-level things that I donā€™t need a library for. I can copy off my GitHub, paste it into a different project, it works the same, as long as Iā€™m in a JavaScript environment. Thereā€™s a lot of value there for me.

Iā€™m going to challenge your premise a little bit on this. This is just pushing a little bit here, which is - letā€™s take the example of component-driven development. Component-driven development - the industry has pretty much normalized on that is the most productive way to do UI development. You break things down into components, you subdivide them in that way, and you do that. And there are a slew of different frameworks that allow you to do this in different ways, but the sort of raw platform in JavaScript does not have a good solution for this. Web Components are *beep*. Theyā€™re terrible. Theyā€™re not a valuable way to do this, because one of the key productivity boosts of this component-driven development is thinking about your UI in a declarative manner. And Web Components, for reasons that I do not understand, chose to maintain an imperative programming model.

So where Iā€™m going to this is - if you invest time and energy in learning how to do component-driven development using one of these declarative frameworks, whether it is React, whether it is Vue, whether it is Angular, whether it is Lit, whether it is one of these other things, abstractions on top of the platform, that understanding, that model - while the framework details will vary, and that piece is not as portable, the mental model for how you do that development and do that breakdown is, and it is something that is going to be more valuable to you in moving from project to project, than if you had tried to do everything from scratch without that mental model.

Yeah, totally agree. And thatā€™s the flip side of the coin, that I think understanding fundamentals informs your decision when you need to reach for a higher-level abstraction. Because if you say that everything should be done with the fundamentals, I think thatā€™s going too far into the other direction. And then you say, ā€œOkay, do everything just with vanilla JavaScript.ā€ And then you say, ā€œOkay, well, Iā€™m going to use whatā€™s available on the platform. Maybe itā€™s Web Components, or whatever, but they donā€™t provide good reactivity models.ā€ So then Iā€™m going to build some sort of reactivity system on top of proxies, or something along those lines, maybe reinvent signals, because thatā€™s pretty cool, and thatā€™s the new hotnessā€¦ And before you know it, we have reinvented one of these frameworks that already exist, but youā€™ve created a crappier version.

[14:24] So that is what you want to avoid, but again, I think the knowledge of understanding what it is that these higher-level abstractions are doing at a lower level, and understanding what theyā€™re doing can inform your decision to say, ā€œWhat does my project need?ā€ and does that need surpass a barrier that says ā€œLet me reach for an abstraction, let me not reinvent the wheelā€?

I think it could be argued, Kball, just to continue down this line of conversation, that at this point, a component ā€“ itā€™s like weā€™ve identified a new fundamental of web development with components. Maybe itā€™s higher level than we previously were, but yet itā€™s so fundamental to the way that we do things now that you are actually learning fundamentals, even though youā€™re learning them through this framework proxy.

It depends on the framework, right? Because if itā€™s React, then yeah, youā€™re doing a lot of fundamental JavaScript maps, for exampleā€¦ But if itā€™s something that has a less fundamental templating library, like Angular - but I know Angular has changed a lot, so I donā€™t know if I can say that, butā€¦ And how is ngf tags going to help meā€¦?

Yeah, well, and thatā€™s ā€“ that is actually kind of an interestingā€¦ Thereā€™s a layering of knowledge here, which is thereā€™s syntax, and tactical details of the APIs that youā€™re using in the frameworks, like how those are working, and then thereā€™s an architectural or mental model level. So if youā€™re using Angular, or React, or Vue, or Svelte, or Qwik, or we can go on and on and onā€¦ Like, this has become the norm, and all of these different things are doing it. They have a set of things that are detailed and specific to their framework, but there is also this architectural fundamental, as you say, Jerod, that is the same throughout all of them.

Yeah. Thereā€™s your transferable knowledge right there.

Thereā€™s your transferable knowledge. Exactly. And I would actually argue that those fundamentals are more important than the details of the APIs youā€™re using, whether youā€™re using a bare JavaScript API for file upload, versus youā€™re using something else for upload. So the medium that youā€™re using to learn it in is less important than that higher-level concept. And another place this shows up is programming languages. So for most software development jobs, you donā€™t need to have prior experience in the specific programming language that is there. You can pick that up.

Sure you do. Itā€™s right there on the job description.

I mean, yeah, there are people who do that. But my premise would be, for example ā€“ I mean, there are differences among different classes of programming languages, but if I have a job that isā€¦ Our applicationā€™s written in Python, say, and youā€™ve done Node, or youā€™ve done Ruby, I think you can pick up Python without much difficulty, because the fundamental understanding of how does a dynamic scripting language, that has object-oriented pieces and functional pieces work, is something that you can get in both of those other environments. And that fundamental knowledge is transferable, and you will learn syntax. And there are - and this comes to kind of your point, I think, Austin - some places where there are real differences between these frameworks, and at some level of complexity or at some level ofā€¦ There are some projects where that nuanced difference does matter, and understanding which parts are fundamental and which parts are specific will be important to you at that pointā€¦ But I would argue that thatā€™s a pretty small minority of projects.

[18:10] So what youā€™re saying ultimately is ā€œIt depends.ā€

It depends. And this of course, was ā€“ [laughter] So before we started recording, Kball didnā€™t want to rehash the fundamentals versus framework thing, and then I wanted to avoid the whole nebulous topic too much aboutā€¦ We were in galaxy brain territory of just ā€œNone of this makes sense.ā€

So weā€™re doing both.

Yes, weā€™re doing both. Hopefully, all three.

Weā€™re just trolling you, Austin. This is just a long roll. Itā€™s all this is. I mean, what does this have to do with uploading files, guys?

Exactly. So I do want to finish with thisā€¦ The reason I say that fundamentals are a loaded term is because a lot of people conflate it with the underlying platform, or like vanilla JavaScript, or like the DOM, or whatever. And there is a distinction there. You donā€™t have to understand every single available method on inputs to know how to use them, right? You donā€™t need to understand that HTML can access the camera. Probably not. Rarely ever going to need that. But again, I think itā€™s just a matter of understanding the lower-level concepts, and stopping and just ensuring that youā€™re taking moments in your day to stop and check your knowledge, and check your understanding of what it is that youā€™re doing.

So Kball, to your point about components, and that being the important thing to learn - I think at a lower level you can ā€“ back to the full stack of knowledge thing, you might start at Next, and then move down to React, and component architecture, like Nick was saying, is a fundamental piece of knowledge within or below React. If you take that, you can swap out React, and you can take that fundamental knowledge to Vue, to Solid, to Qwik. And there are going to be some differences, and understanding those differences is the implementation detail of the next level up.

Not to dive into that too much, but then those differences also can shape how you think about those individual components.

For example, if youā€™re trying to avoid massive rewrites within a virtual DOM environment, you might break up into smaller components. Whereas maybe another framework better manages that, and doesnā€™t require you to do that, so you donā€™t have to break it up nearly as far.

Yeah. Itā€™s interesting, because Iā€™ve shifted to working with Solid a lot lately, and the way it renders things is totally different than React.

And that is fundamental knowledge you need for that tooling, because then you canā€™t build good applications without understanding what itā€™s doing below, under the surface.

So it is fundamentals all the way down.

It is, it is, ultimately.

So letā€™s bring this back to maybe a more practical, tangible topicā€¦

No, no! Quick, someone interject something. Basketball, yes. [laughter]

Well, I am kind of interestedā€¦ So you have dug into file uploads, and so I think that is a useful thing. Can we contrast doing bare file uploads using platform technology only, versus using a library that provides some abstractions around file uploads, and use that as a way to illustrate what are the fundamental pieces that are not specific to the, as you said, input APIs, and things like that, that actually do play out whether youā€™re doing this with bare platform, bare HTML and JavaScript versus something a little higher-level?

Yeah, absolutely. So this conversation is kind of based on a blog post series that I write, and I donā€™t think that my writing is particularly unique or special. I think that thereā€™s a lot of excellent writers out thereā€¦ But I think a gap that I try to fill is in all of these blog posts that I read, that kind of miss either the next logical step, or just like go around some lower-level concept that is necessary, or you want to understand how this stuff actually worksā€¦ So I try to provide this fundamental knowledge in what Iā€™m writing. At least like the one layer below.

[22:01] So to get into this sort of topic, I think the place to start with file uploads for me - I generally lean more towards the frontend. You could start in different areas, but I always like to start on the frontend, because thatā€™s what I see. And the most basic fundamental thing on the frontend is HTML. And even that, if weā€™re talking about file uploads, you canā€™t just talk about file uploads by understanding HTML, you have to understand the fundamental thing below it, which is HTTP. Because you need to make an HTTP request in order to upload a file. Yes? Cool? There so far?

Youā€™re now going to teach us HTTP. Go!

Yeah, no, Iā€™m not going to teach you HTTP. I probably donā€™t understand it well enough.

And to understand that, you have to understand TCP/IP.

And to understand that, you have to understand Ethernet.

Thatā€™s the level that I probably donā€™t understand.

[laughs] There you go. So we all have our limits, right?

Yeah. And again, you donā€™t need to understand everything all the way down, but you do need to understand the things kind of at the lower level, from the point that you work in.

Right below you.

Yes, the next level below at least. So with HTML, to send a file, you need to do a number of things. In HTML, the communication language between computers on the web is HTTP, and to send a file with HTTP, you need access to the file contents. You need to modify the request header to send a post method, and you need to set the content type to multi-part form data.

So if weā€™re at the level of HTTP, can you break down - what is the difference between a get and a post, at least? Because I think that is actually a key thing and something that we sometimes paper over with tools like GraphQL, and start using in strange ways.

Yes, thatā€™s a good point. I think that thereā€™s going to be probably some areas of knowledge that yā€™all feel free to fill in. As I understand, an HTTP request is basically a protocol that you constructā€¦ I want to say it kind of looks like a string of text that a computer can send to another computer, but the way that string is constructed is important. So you define the HTTP headers, the version, and then the method, and the method can be I think one of either seven or nine different options. But with just HTML, HTML really only has access to two methods - the get request and the post request. And the get request is generally saying, ā€œIā€™m trying to receive some dataā€, or ā€œIā€™m asking for data from the other computerā€, and a post request is saying ā€œI have some data that I want to send to youā€, to another computer.

Yeah. So the biggest difference with that, falling within those params is that a post can also have a body on it, whereas get doesnā€™t.

Yes, it is a ā€“ I donā€™t know if itā€™s against protocol to include a body on a get request.

It would often be ignored.

The only way you can pass data, or get requests ā€“ well, there are two ways. Thereā€™s the headers that youā€™re sending, and then thereā€™s params, which are part of the location, the URL string that you are sending this to. So any HTTP request youā€™re sending is going to a place, conceptually, a virtual place, but that is a URL, and it has a set of data passed as params, which some backends will treat as giving you more information about what place youā€™re looking for. Like old school WordPress, youā€™d pass a parameter and that would tell you what page youā€™re looking for, things like thatā€¦ Others will just treat it as essentially modifying, but thatā€™s the only way you can pass data along. And the headers are relatively constrained, the params are not. You can put whatever you want in there.

But you could also pass some data ā€“ a REST API might pass some data via the URL itself. So you have the URL location, and then you also have the query string at the end.

Yeah, that location can be interpreted as data by the server. But you canā€™t, for example, put a blob of binary data or something else in that.

But with a post body, you can.

You can do that. Yeah, and then you also need to set some content type along as the HTTP headers to denote what type of content youā€™re sending. So this is really a fun conversation, because now weā€™re actually getting into practical knowledge, that I think ā€“

Letā€™s just start the show right here.

Yeah, exactly. [laughter]

Right, right, so thatā€™s a problem. Weā€™ve got to Figure out a basketball, or soccer, or ā€“

Some sort of way to derail Austin before he gets into detailsā€¦

Can we do field hockey?

Okay, so field hockeyā€¦ So whatā€™s ā€“

This is considered a pivot, I think, but in basketball if you lift that pivot foot, itā€™d be traveling, and ā€“

Thatā€™s right.

And Nickā€™s exhausted his knowledge right there. Then what happens, Nick? [laughter]

I have the Wikipedia page open for basketball right nowā€¦

Okay, but the ā€“ thatā€™s brilliant. Kball, Iā€™m glad that you took us down that, because I didnā€™t touch on that lower-level HTTP protocolā€¦ Which is funny, because itā€™s redundant to say HTTP protocol, but I do it every time.

Thatā€™s right.

Because itā€™s the last P, isnā€™t it?

It is the last P.

The last P stands for protocol.

Hypertext Transfer Protocolā€¦ Yeah. Who cares? Okay, so weā€™ve kind of established that fundamental knowledge of HTTP, right? And then you get into the browser and the language of the browser, the sort of base-level language of the browser is HTML. And with HTML, you have access to making get requests using links, you can make get requests using formsā€¦ But thereā€™s only one way to send post requests, which weā€™ve established is necessary to send a file; the only way to make a post request is with an HTML form. And we also established that you need to have access to the actual fileā€™s contents, and you have to send the multipart form data HTTP header, right?

So with the browser - the browser provides a sort of sandbox ā€“ itā€™s an application. Itā€™s a computer application that communicates, and it translates HTML into a visual layout of a website, and it constructs HTTP requests for you, without you necessarily having to know whatā€™s going on. But the browser sort of works as a sandbox environment that doesnā€™t give access to website developers, doesnā€™t give them access to user file systems. Like, you canā€™t build a website that accesses its userā€™s file system.

A little asterisk there, because there are some experimental technologies about file system access, and thatā€™s sort of an aside, because itā€™s a limited file system accessā€¦ So before anyone kind of tramples on me, letā€™s just say ā€“

I was so close.

I know. You donā€™t have access to like the full file system. Any file in the userā€™s computer, right? In order to do that, you have to have user interaction where the user can select the file, and in order to do that with HTML, you can only use an input with a type of file, right? So now we have access to the file, we have a form to construct a post request, you set the method on the form to post, you have to set the content type to multipart form data, which you do with the [unintelligible 00:28:42.17] type attribute on the form elementā€¦ And bingo-bango, you can send files with HTML. Thatā€™s kind of the bare minimum. Cool?

And so what thatā€™s doing under the covers is that file as a binary blob is getting sent along as the post body in HTTP. So then anything thatā€™s responding to that, something on a server somewhere, is able to access that blob.

So itā€™s also important to point out that if you leave off the method, browsers default to using a get request. So it just adds data from the form in query string parameters. If you leave off the [unintelligible 00:29:21.16] type, then browsers will send the body of that file, or itā€™ll append the data from that file either in a get request to the query parameters, or in the body as just a string of the file name, I believeā€¦ So if you want to have access to the actual binary data from the file contents, it has to be multi-part form data. And it doesnā€™t actually send it as a one blob in the request, because a file can be a large thing, so it can exceed the amount of data that can be included in a single request. So what the browser will do is send that as a stream of chunks of binary data. So like send little bits at a time.

[30:06] Then the next question that people ask is, ā€œWell, yeah, HTML is cool. Iā€™m glad I can do thatā€, but everyone wants to do the thing where the page doesnā€™t refresh, because when you just use HTML, you hit submit the form, the formā€™s gonna reload the page, and whatever. We want to provide a nicer experience for users using JavaScript. So then whatā€™s the next question with JavaScript? Well, you still have to do the same things. You technically donā€™t need an HTML form, because now you can construct an HTTP request using XML HTTP request, or fetch. Most likely fetch. So fetch can still construct the HTTP request, provide the URL, set the method to post, set the correct HTTP headers, and then you still need access to the file system, which still requires at least a file input, because you still need user interaction to select the file, because the browser doesnā€™t have access to the file system.

Thatā€™s worth actually ā€“ I think thatā€™s a fundamental thatā€™s worth diving into a little bit more as well, which isā€¦ Thereā€™s a set of things that you can trigger on your own with JavaScript. And thereā€™s a set of things that require user interaction. And thereā€™s, I think - I donā€™t know off the top of my head what the list of things are, but thereā€™s a set of things where basically you have to beā€¦ What that plays out in is you have to be ā€“ well, either itā€™s handled by the browser, like attaching a file via an input, or you have to be inside of a user-generated event handlerā€¦ An event handler for a user-generated event for it to work at all. Do you happen to know offhand what is that list of things? Or maybe even better, whatā€™s the conceptual model behind when thatā€™s going to be true?

Dang, you put me on blast.

On the spot. [laughs] No, no, no, I could not tell you. I probably couldnā€™t even get close. But yeah, there are some things that can only happen within user-generated events.

Most of the time once you try to do it, itā€™s logical. Youā€™re like ā€œOh, this makes sense.ā€ And itā€™s usually security and privacy. Youā€™re like ā€œWell, Iā€™m not going to just be able to send myself arbitrary files, because you loaded my webpage.ā€ But once you interact, you can select the files, and now I have access to that thing. That just makes logical sense with regards to privacy and security and user intent.

If you want to know all the things you can do without any interaction, visit themostannoyingwebsite.comā€¦ Ferossā€™ website that will trigger every crazy thing that JavaScript can do, all at once, and youā€™ll have to actually probably reboot your browserā€¦ So donā€™t actually visit that, but if you want to, give it a shot.

You should do it while weā€™re recording, Jerod.

Yes, I should do it right now, and then Iā€™ll just disappear.

Yeah, yeah. Shout-out to Feross, actually, because Kball, thatā€™s a really good point that touches on, again, fundamentals that really lean into security principles. Because security is one of the pillars of web development, or requisite knowledge, and understanding what can happen ā€“ like, thereā€™s a lot of dark patterns around tricking the browser to believe that a user did something, or attaching events to certain user interactions to then do things that would not otherwise be intended, like maybe clicking on hidden or a not visible thingā€¦ I donā€™t know. We donā€™t have to dive into it, but Iā€™m glad that you brought that up. Again, fundamental knowledge that apparently I need to fill in.

I think it is key to realize, browser vendors are basically trying to protect their users, and make sure that you are doing things ā€“ when you do things to them, it is because they asked you to. And so anything that is going to involve touching something from their computer, doing something that feels intrusive, launching a pop up, all these different things - browsers will try to make sure that youā€™re doing this in response to a user event. And that has been an evolution over time, as people like Feross found all of the edges and the places where you could do things to a user that they maybe didnā€™t wantā€¦

[34:03] And browsers have evolved over time. I know weā€™ve had conversations with folks who used to be in ad tech, where they were like ā€œIt was an arms race between us in the browsers, where we kept trying to figure out how can we do a pop over, a pop under, a be more in their face, and the browserā€™s kept being like ā€œNope, nope, nope.ā€ But that core concept of fundamentally the browser is attempting to defend the users from malicious programmers is, I think, a key way to think about what it does and does not enable, and how it enables things when it enables them.

And I donā€™t think that this is a was thing. Itā€™s still actively ongoingā€¦

ā€¦and itā€™s personified perfectly, I think, in the recent Apple Vision Pro announcement, where that has a Safari browser on it, but it is explicitly going to protect the user from where their eyes are gazingā€¦ Because it can know exactly what youā€™re looking at in Safari, but itā€™s not going to give that information to the user, because thatā€™s so much more private than where your mouse is, for example.

Wait, stop. Tell me more. How does that - like, Apple, Safari in there wonā€™t give you access to it?

To user eye-tracking.

But isnā€™t that how a lot of things are triggered?

Itā€™s at an OS level which wonā€™t leave the device.

So web developers building websites in Safari on the device wonā€™t have access to eye tracking technology. However, the operating system does.

Yeah, so theyā€™ll get access to higher level events, like ā€œOh, this is a synthetic click eventā€, or whatever, that may have been triggered by eye-tracking, and I was looking at it, and whatever, but the underlying eye location data is not going to show.

Itā€™s all about consent.

Yeah. So we consent to you continuing your tutorial.

Awesome. So weā€™ve essentially sent things from the client side. And I think outside of the GraphQL stuff - because thatā€™s outside of the scope of my knowledge, uploading files with GraphQL; I donā€™t know - probably at some level itā€™s HTTP, so youā€™d have to do the same thing. Thatā€™s the lower-level ā€“

I mean, everything in GraphQL is a post, so itā€™s in some ways a shorter gapā€¦

Yeah, yeah.

Though having the combination handler that understands both GraphQL and the file - thatā€™s anotherā€¦ A special snowflake.

Yeah. Oh, GraphQL error handlingā€¦ Itā€™s awesome.

Everythingā€™s 200. Itā€™s good.

Everythingā€™s a 200. Great.

Alright. So we made that reference. Weā€™re talking about fundamentalsā€¦ Whatā€™s the 200?

Oh, goshā€¦ [laughs] A 200 is a status code that says everything is ā€“ I think itā€™s a status code for receivedā€¦

Okay, I think.

Itā€™s okay, isnā€™t it?

Or okay? Yeah, that the request went okay.

ā€œYou sent me a thing, I got it. Itā€™s all good.ā€

Even if itā€™s not good, becauseā€¦ GraphQL.

Yeah, actually that was a ā€“ the status codes is a sort of a milestone in my career, where I recognized that I understood the fundamentals of HTTP better thanā€¦ You know, I started as a frontend developer, got into backend development, and then did a whole bunch of mistakes on how to handle communication between the two, because I was trying to do things at a higher level, and send things in like JSON responses of telling the client if the request was okay or not. And then you realize that no, if you just follow these decades-old protocols that all of this is built on top of, and you kind of opt into those, you now are doing things using tools that have been built over decades, and have a much larger ecosystem, therefore a much larger knowledge base; youā€™re not building bespoke things that only apply in your case, and everyone else that joins your organization is going to do things in a standard wayā€¦ And then when you move on, you can take those standards with you. Itā€™s great.

Well, and that points back to your original point of the value of fundamentals, right? You donā€™t end up reinventing the wheel. You donā€™t have to redo all of these things, and thereā€™s a lot of stuff that you get for free at the platform level, if you follow platform-defined semantics.

[38:01] Yeah. Okay, so we sent out files from the frontend, right? Now, the lot of tutorials will stop thereā€¦ Not this guy. We want to move on to the backend, because thatā€™s the other side. Like, anyone can just send files anywhere, you know? But I want to receive them. So this is a JavaScript podcast, Iā€™m a JavaScript developer, weā€™re building applications in Nodeā€¦ You might use Deno, or Bun, or something, or one of the other edge runnersā€¦ I donā€™t know. But anyway, a lot of these platforms are adopting Node standards, which is great. We want portable code, thatā€™s awesome. And the way to receive file uploads - again, touches on HTTP; you know that youā€™re gonna be receiving a post request, you know that that header, the content type header is going to be multi-part form data, so you need to tap into that. And again, because we discussed how the browser is sending data one thing at a time, you have to be aware of that data coming in at different events. So in Node.js, a lot of people are familiar with Express, but even at the lower level, Node gives you access to an event handler system with the incoming message request object. So you can say request.on, and then listen to the data event, and that data is going to come in one little chunk at a time from what the browser is sending. Itā€™s not all going to be in one massive payload. Because if weā€™re dealing with a 10-gigabyte file, too much for one HTTP request, right? Or one chunk.

So that becomes interesting, because then you have to do something with that data. And the data comes in as chunks of binary code, which JavaScript doesnā€™t really know how to deal withā€¦ And so JavaScript has this concept of buffers, which are also confusing, because when you log into the console, itā€™s just something that says, ā€œGreater than buffer, a bunch of string ā€“ of numbersā€, and then close bracket, right? And youā€™re like ā€œWhat the heck is this?ā€ So you can convert it to a string, then you can start to see the little bits of string representation of that data, and thatā€™s niceā€¦ But you still have to do something once it all comes in. So the naive approach is to say ā€œTake this chunk of data, convert it to a string, push it into an arrayā€, and then as each chunk comes in, you push all of those things onto an array. And thereā€™s a problem there, because eventually, that array is going to take up more and more space in memory, which is fine for small HTTP requests, but because files can be quite large - letā€™s say a 10-gigabyte file, if youā€™re running on a server that has two gigabytes of memory, itā€™s going to exhaust those resources. And everyone thatā€™s had to create file upload scenarios probably has run into the server crashing, because it ran out of resources.

Well, and this is highlighting another fundamental concept, which is streaming data, versus serialized data.

Exactly. And the streaming stuff - for anyone listening, if this is your first experience with streams, itā€™s really confusing. I donā€™t even want to try and tackle explaining it, because it took me probably like 10 times of reading the same thing, reading different things, doing reading, versus video, versus presentations, before I actually had a decent grasp. So I do want to stop and say, if you are coming into streams your first time, be kind to yourself, and understand that a lot of people have problems with it, and thatā€™s okay. Youā€™ll get around to it.

I mean, I think there is a fundamental sportsball metaphor that we can make hereā€¦

Sure, letā€™s do it.

Letā€™s do it. Sportsballā€¦! So imagine that you have three people in spread out from each other, and you have a pile of soccer balls. You have like 10 soccer balls, and they start with one person. And you want to pass those balls along your lines, through a fireman style, one, two, until they get to the end. A stream would involveā€¦ A person starts with ā€“ has 10 balls, they kick one to the middle person, that middle person immediately kicks the ball forward. And so as you transfer these balls, the person in the middle, as soon as they get a ball, theyā€™re kicking it forward. They donā€™t have to have a lot of balls all at once.

[42:03] A buffered approach, or a serialized approach would mean you kick all the balls to that middle person, they collect them all, until they have their 10 balls. Then they turn around and they kick those 10 balls off to the next person. And so you end up with everything together at each step, bunched along. So bringing this back to programming, streaming is essentially saying, ā€œHey, Iā€™m only dealing with a small chunk of data at a time, and I do whatever processing I need to do on it, and then I pass it on and I forget about it. Iā€™m not worrying about it.ā€ And that is fundamentally more memory-efficient, because youā€™re only storing in-memory that small amount at a time.

Now, thereā€™s things that are harder to do like that, right? Like, if I wanted to analyze an input file, some things I might be able to do easily. Say Iā€™m counting the number of aā€™s in a string - okay, I keep one counter. Look at this chunk; do I have any Aā€™s? I add them on, I pass it along, whatever. But if Iā€™m doing like a whole image analysis, thatā€™s not going to work as easily in a stream-based manner. So there are problems that are easier to do in a streaming way that not, but fundamentally it has to do with ā€œAm I storing up all the data as itā€™s coming in, so I can look at all of it at once? Or am I looking at each piece and then passing it off and forgetting about it?ā€

Yeah, thatā€™s a good point. I work at Akamai and we have edge compute technology, and one of the things that you might want to do with edge is you receive HTML from the origin server, and then you might want to do some sort of rewriting of the HTML, and then send it to the clientā€¦ But because youā€™re dealing with chunks of data at a time, what happens if you want to modify, letā€™s say, the body tag, and one chunk of data comes in with open bracket BO, and then thatā€™s it, and then the next chunk has DY, close bracketā€¦ You canā€™t ā€“ itā€™s really hard to tackle string replacement across chunks.

But anyway, so going back to the streaming thing, like, Kball, you were saying - that does introduce streams, right? You want to avoid writing files and potentially large payloads into memory because of that. So streams are the solution, where you open up a file write stream, and on the server thatā€™s running your application, as you receive each chunk of data, you maybe convert it to a string, and then you write it to the file system, and then you release that memory. And then the next chunk of data can come in, or multiple chunks can come in at a time, but youā€™re writing them to disk and then releasing them.

Now, Kball, this concept comes back to one of the points that you made about abstractions. Weā€™re talking about the very low-level Node APIs that are available. And at this point, weā€™re talking about ā€“ weā€™ve basically described creating your own server, HTTP server, and creating your own file or routing system and file handler. You donā€™t need to do that. Weā€™ve had Express forever, Fastify is awesome, and they provide these higher-level tools. That makes sense, because I think you should understand how Node does thisā€¦ But at this point, these are tried and true, and Iā€™m not going to improve upon them. And they provide all the tools that I do need, and that I will need. Itā€™s worth it to opt into those.

Totally. Well, and the concept of streaming data is transferable, right? You can write streaming versus batched operations in Python, you can do it in Go, you can do it in Rubyā€¦ That core concept is going to be transferable regardless of whether youā€™re learning it first in Node, first in Node with an abstraction, or what have you. Wrapping your head around that distinction is where the fundamental transferable knowledge is.

Yeah. So those abstractions are good to know. In the article that I wrote, I go into ā€“ basically, Iā€™m building this application in Nuxt.js, which is a very high-level abstraction. Nuxt.js is a metaframework of Vueā€¦ But because I know Iā€™m going to deploy it to a server thatā€™s running Node.js, I like the fact that Nuxt.js gives me an escape hatch to tap into the lower-level system of Node. And once I have access to Node, then I can write a middleware, or do things in the Node way, that I can take that chunk of code and bring it in and use it anywhere.

[46:06] So I can write a file stream handler using the Node ecosystem that will work in Next, in Nuxt, in just plain Node, whatever. And that also gives me access to the Node.js ecosystem, which is abundant, with tools like file system stream writers like Formidable.

So a library that I really like thatā€™s part of the Node ecosystem is Formidable. It was written a long time ago, where everything was done with the error handling upfront, the callbacks approach, where you get the error message, you check if thereā€™s an error, whateverā€¦ And so you can take that and you can write essentially a function that passes the underlying incoming requests from Node to Formidable, and then you have your own little well-tested, established tooling that can then live and be put into the Nuxt ecosystem, because you have a good understanding of where those boundaries are of what is the Nuxt responsibility and available tooling, what is the Node responsibility, what is the Formidable responsibility, and how do they interact with each other and interface with each other. Yeah?

You guys are nodding your heads, but this is an audio show. Please confirm audibly, yes. Iā€™m here to say yes. Nick?

Thank you. Kball?

Iā€™m already talking enough. Yā€™all can handle this.

[laughs] No, I really appreciate that. Kball, youā€™ve been doing a great job of checking me, so I appreciate it.

So yes, I think thereā€™s something more than just understanding the lines, which is that a well-designed framework or metaframework will actually give you seams where you can insert things from lower-level abstractions. They will make it easy to do. I think this is something ā€“ if anybody out there remembers the Ruby on Rails ecosystem from 10 years ago, Rails was bad at this when it started. It was ā€œWeā€™re gonna make a choice for you, and itā€™s hard for you to sub in your own choices.ā€ And over time got much, much better and cleaner, of saying ā€œOkay, hereā€™s how you sub in your own choice. You can alternate ā€“ā€, and it kind of created these seams where thereā€™s like a little thin abstraction layer around an underlying piece, or technology.

So I think in the JavaScript ecosystem right now, when you look at something like Nuxt, theyā€™re doing a reasonably good job of that. Theyā€™re like ā€œOkay, youā€™re running something on the server; hereā€™s where that goes, and within that, you can plug in anything from Node you want.ā€ And so theyā€™ve made it relatively straightforward to follow your understanding of fundamentals, and like cut in the different pieces that you want, where you want it. I think that that is something to look for also when youā€™re adopting these higher-level abstraction, is ā€œDo they provide those seams, those cutpoints for you to come in and inject changes that operate at a different level of abstraction?ā€

Which is hard to do.

Because if they donā€™t, thatā€™s where you paint yourself into a corner, and then all of a sudden ā€œI need to do something that they donā€™t offer, and I donā€™t have access to the underlying ā€“ the thing that theyā€™ve abstracted over, and now Iā€™m stuck. Or I have to find something else.ā€ Thatā€™s the worst.

Okay, so I think that that pretty much covers the technical side of things. But I donā€™t think that that paints the whole story of what to deal with and what to think about when youā€™re allowing for file uploads, because - well, first off, it doesnā€™t look at the holistic architecture view of building applications these days, which could be distributedā€¦ So if you are building a distributed system, and you are receiving files and writing them to disk, but you have multiple computers that are processing your requests, all of a sudden itā€™s really hard to track down where those files are being stored. Which server, which location, and stuff like that.

Which is why you stream them to S3 or something equivalent, right?

[50:02] Exactly. On top of that, on top of the trouble with distributed systems is what happens when you start receiving large files and you have a server thatā€™s maybe like a lightweight server, and totally enough for your application to run, but it has smaller disk space, and all of a sudden, large files are taking up a lot of the disk space, you run out of disk space, you have to upgrade your server. Now, if you have to upgrade your server because you want to add more compute, youā€™re getting sluggish, or you need more CPU - excellent reasons to upgrade a server. If you are running out of disk space, terrible reason to upgrade a server, because you may end up doubling the cost at a not very cost-effective way of essentially doubling the disk space. And before you know it, youā€™re running out of disk space again.

So Kball, you mentioned S3ā€¦ S3 stands for Simple Storage Solution, I think. S3 was created by Amazon, along with their service. They created their own protocol for storing files, adding files, managing files, deleting, updating, whatever. And itā€™s been great. And that protocol has been adopted by a lot of other file storage solutions partners. I do work at Akamai, I feel like I should interject that we have an object storage service that we offer, that is S3-compatibleā€¦ And the article that I write, again, speaks to fundamentals. Yes, I work for a company that offers object storage solutions, yes, in my examples I use thatā€¦ But everything that I write, I try to do, again, at a fundamental level, so you can take this and apply it to anything that is S3-compatible. So anything that uses this protocol, take it and put that knowledge where you want it. Very important to me.

But one of the big factors for object storage, S3-compatible object storage, is that they are designed for these files. So theyā€™re highly available, easy to access, they store it in one location, and theyā€™re very, very cost-effective. So for the same price that you would get 25 gigabytes more on a VPS to upgrade your VPS, you can get 250 gigabytes, at least with Akamaiā€™s cloud computing services. So thatā€™s a 10x difference. And that applies ā€“ and again, that would be across different service providers. Iā€™m only saying the numbers that I know off the top of my head, butā€¦ Not trying to turn this into a sales pitch.

So in this case, the core concept is object storage, which ā€“

Yes, object storage, whoever you go.

ā€¦is essentially like the distributed version of file storage, right?

Itā€™s like, ā€œHereā€™s a location, that leads to a file, that has some set of permissions.ā€ And you can even do directory structures the same way you might in a file systemā€¦ But is there anything more to it? Is there any other concept? Or is it ā€“ if somebody is used to thinking about files, is this just files in the cloud?

I mean, I would say basically yeah. If youā€™re looking at object storage solution providers, I would encourage looking for S3-compatible object storage. I think most object storage providers are S3-compatible, because itā€™s become such an industry standard. And the benefit there, again, is you have portability. You can write your code in one way, and as soon as a different provider, a different S3 provider, or S3-compatible provider offers something that is a deal-breaker for you, or maybe the current provider that youā€™re using changes pricing, or planning, or how youā€™re allowed to access your data, and you want to transfer it - cool; you donā€™t have to change your business logic of your application. Thatā€™s really important to me.

Any more on object storage? Itā€™s highly cost-effective, regardless who you use, and also very simple. And the key is - I wonā€™t get into the weeds on the implementation details, but the interesting thing when we bring it back to how you move your files from writing to your server, your application server, to object storage, matters, because thereā€™s a couple of different ways. And the question is how do you get those files into the object storage provider? And I think almost every piece of content that Iā€™ve found out there suggested using whatā€™s called signed URLs, where essentially your client makes a request to your backend, the backend serves as a proxy for the object storage hosting, and the backend will make a request to the object storage provider for a signed URL. A signed URL is essentially the location where the file is going to be uploaded to, but it can only be uploaded to by the person with that signed URL. Itā€™s basically a URL that includes some sort of authentication mechanism, or authorization mechanism, as well as some expiration date.

[54:41] So your backend can provide that to the frontend, the frontend can take that, then upload the file directly to the object storage provider. It doesnā€™t have to go through your backend at that point. And they can get the file there. Thatā€™s what most people say is the best way, and it is a decent way. I personally prefer building my applications with progressive enhancement built in. You cannot do that, I donā€™t believe, in a progressive enhanced way, because I believe the only way to use a signed URL is sending a put request, which HTML cannot support. So that breaks. But there is a different way, and thereā€™s a couple ā€“ Iā€™ll explain the other way first, and then maybe we can look at the pros and consā€¦

The other way is you can send the file directly to your proxy service. And again, instead of streaming to writing the file to disk, you can pass that stream along and upload it as a stream to the S3 storage location. The big downside there is that - well, thereā€™s a couple of downsides. One, you are basically paying potentially double for that file, because you are going to pay For the file being stored in your object storage, but you may also be paying as your server, which serves as a proxy, is receiving that file - thatā€™s ingress cost; and as it streams that file out to the S3 provider, that is egress cost. So because itā€™s passing through this proxy, you may be paying more, because of the traffic thatā€™s being run through. Your mileage may vary, check with your stackā€¦ Again, you can take this to anyone.

The other sort of advantage or disadvantage is because that file is passing through your proxy servers, now you have more control over what to do with that file. For example, if you wanted to do file optimization, if itā€™s images, you canā€™t do that if people are uploading directly to your object storage provider, unless you also pull it down, in which case youā€™re going through your server anyway. The other sort of downside is, once again, because that file is passing through your proxy server, what happens if someone wants to upload malware? Now, if they upload malware through a signed URL, itā€™s going to go to the object storage. Itā€™s kind of in quarantine there as long as you do some sort of malware scanning on itā€¦ But if youā€™re passing a file through a proxy service, that malware could potentially be into your system, and then pass through. So then the question becomes, ā€œHow do I deal with malware on my uploads?ā€ And once again, I think that this is the advantage ā€“ as much as it might be a disadvantage, itā€™s also an advantage of passing files through your service, because it gives you an opportunity to actually scan those files for malware. Any questions here so far before I continue into malware?

Iā€™m waiting for the malware apart, man. Well, how are you scanning these things?

Okay. Okay, so dealing with malware is interestingā€¦ OWASP has a whole cheatsheet that I can link to, or is in the blog post that I link fromā€¦ OWASP has a whole cheat sheet on how you deal with malware in the context of file uploads, and itā€™s pretty interesting, because once again, this sort of library, Formidable, that I had mentioned before - itā€™s really awesome for dealing with files, because it handles all of this stuff that you would want to do sort of for you, or at least provides an easy API interface for doing thatā€¦

One of the things is you want to ensure that ā€“ well, essentially, to deal with files or malware, you kind of have to have the entire content of the file. So remember how we said you could stream a file from the client to your proxy server, or to your backend, and then pass that stream along to the object storage location? If you want to address malware, you may not want to do that; you may want to go back to writing it to the disk on that server, because you need the entire contents of a file to know whether itā€™s malware, or to scan it for malware.

[58:23] Right. But you could just stream it to like temp storage or something, and thenā€¦

You could stream it to temp storage, or what you could do ā€“ yeah, and you could have a temporary sort of quarantine folder on disk, that is the temporary storage, and then pass that to object storage again, which is probably how you would do this if you went this approach.

Or have like another bucket thatā€™s like your temporary object storage bucket, if youā€™re thinking distributed systems. Those disks ā€“ I mean, local disks just destroy distributed systems when it comes time to actually distribute themā€¦

Yeahā€¦ I mean, yeah, you could have another object storage location, but if you want to scan it, youā€™d still have to pull it down to some server, and then scan it thereā€¦ So it would kind of be redundant. So I think what you might do is

Well, it depends on how long these processes take, right? You could throw it in a background queueā€¦

Thatā€™s true.

ā€¦and you could have a bunch of workers that are working off the backend to just scan those thingsā€¦

Thatā€™s true. You could have you could have a separate backend just for scanning files.

But I donā€™t know how long it takes. That maybe helps inform this discussion.

Thatā€™s a good question. So a lot of the OWASP recommendations are like obscuring file names, or not using the file names from the client, because that could use disallowed characters, checking file types, checking content, and the [unintelligible 00:59:38.03] or the MIME type of the file, and sort of like filtering that way, preventing certain file sizesā€¦ So if you have a system thatā€™s like only scanning PDFs, you donā€™t need gigabytes of dataā€¦ Versus an application that is going to be doing like video editing - you may need to support multiple gigabytes. So this is where you get into like more bespoke decisions around what your application is.

But ultimately, you can have malware scanners running. Like, basically receive the file, write it to disk, run a scan on it, but that can take a long time, and therefore you canā€™t really build a system that does that within the request/response cycle, because a request and a response has to be quick. Basically, just the time it takes to upload it, and then I need to know that itā€™s there. So you would want to build some sort of background process that then requires a database, that keeps track of all of the metadata about the file: the file name, where it exists on disk, what the MIME type is, and then some sort of other information of whether it has been scanned or not, and then youā€™d have some sort of background process that says, ā€œOkay, check the database, look for the files that have not been scanned, pull them off of their location, download them, scan them if theyā€™re good, put them back, mark them as greenā€¦ā€ And then this kind of comes back to ā€œAre you going to be presenting these files back to a client somewhere?ā€ Because if you are, you want to be able to upload those files, you want to be able to tell people that their files are uploaded, and you have a record of them; so you might show that to people on the frontend. But if they have not been scanned, you want to show them the metadata, but not provide a link to download that, because you donā€™t want to ā€“ even if you protect your system, even if malware gets into your system and you have a way of quarantying it ā€“ quarantining itā€¦ [laughs]

Corinthiansā€¦?

Something in thereā€¦ You know what Iā€™m saying; protecting it.

Are we doing Bible verses?

Yeah. [laughter]

1 Corinthiansā€¦

You want to be able to protect your system, because you donā€™t want to ā€“ you donā€™t want to give a file access to maybe reading your database credentials, or whatever. But you also want to ensure that your system can be trusted by your users. So you donā€™t want to become a source of distributing malware.

[01:01:50.21] Well, and you can see this example anytime you upload a file that is large enough that itā€™ll take substantial time to a social network, or YouTube, or something like that. Theyā€™ll say, ā€œOkay, this is uploaded. We are scanning it.ā€ Then once itā€™s scanned, itā€™ll go live. And the big guys make it real fast. They do what they can to make that happen really quickly. But it can take some time.

I think to Jerodā€™s earlier point, by the time youā€™re doing all this tracking and background tasks, itā€™s going to be simpler if you build distributed-first. So you put it someplace in S3, you track it, then you have whatever your background cue is; theyā€™re the ones pulling it down locally, and they donā€™t have the latency worries. So they can just pull it down locally, do all the scanning that needs, mark it off wherever it is; you donā€™t have to do quite as much bookkeeping, in that sense. So yeah, I think once youā€™re thinking distributed systems, object storage all the way.

For sure. Hopefully ā€“ I mean, I think that kind of tied up the whole solution. We did HTML, JavaScript, Node, metaframeworks, storage solutionsā€¦ We didnā€™t talk about delivery, which is a CDN to basically make sure that files that are being downloaded to people in Tokyo, or Toledo, or Toronto - they all get them as fast as possible.

I should just mention that CDNs are basically like a globally distributed network of connected computers that can be used to automatically identify traffic, and theyā€™re sort of distributed as close to users as possible. So they can route traffic to the nearest node in the network from where the user is coming from. And theyā€™re useful for storing things like static assets, so that people donā€™t have to request things all the way from an origin or an object storage location. And they also these days are a lot more advanced; they can offer features like web application firewalls, DDoS protection, and other awesome security features.

So coming back to then your file example - after somebody has uploaded this file, theyā€™ve stuck it in an object storage, CDN basically will ask that object storageā€¦ The first time somebody requests that file, theyā€™ll ask the object storage, theyā€™ll get it back, and then itā€™ll store a copy of it all out on these edge nodes, which are computers all over the world close to where users might beā€¦ And then when future requests come in, following all the cache protocols, and whatever your cache timeouts are, itā€™ll get it just from that local edge, not even go back to that object storage.

Kind of yes. I think most CDNs work with a poll feature instead of a push feature. So when you upload a new file, you donā€™t push it to the entire network. Itā€™s just when a user that goes through a specific node, that node, the first time that request is made, will pull the file from its origin, which might be an object storage location, and then store a cached version. And then any other user that passes through the same node will get that same file stored and delivered at like a fraction of the time.

Got it. So itā€™ll go back to the object ā€“ you know, Akamai has thousands of nodes out there.

So I might get thousands of hits back to my original object storage, but only one for each location.

Yeah, so it could reduce some of the bandwidth needs for your origin server as well. Because if youā€™re gonna get thousands of hits from users around the world, youā€™re going to receive those the first hit, no matter what. But then if theyā€™re coming from the same location, you would only get one hit, and pay for thatā€¦ In theory.

In theory. Everything in theoryā€¦

I was struggling to find a sportsball reference for that. Itā€™s like tennis, the guy standing at the net, and you hit the ball over, and heā€™s like ā€œNope.ā€ Hits it right back to you. But it doesnā€™t work quite right.

Yeah, I canā€™t ā€“ I donā€™t have a good one. I watch roller derby, soā€¦

Thatā€™s alrightā€¦ Another episode is in the can. And just like coaching kids in basketball, understanding JavaScript file uploads is just mastering these fundamentals. So whether itā€™s a three-point shot, or a three-gigabyte file uploadā€¦

ā€¦practice makes perfect. So keep coding, and keep coaching, and weā€™ll catch you next time.

Changelog

Our transcripts are open source on GitHub. Improvements are welcome. šŸ’š


About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK