Darcy: what is more interesting to me is actually finding a way to operationalize open source in a new way that creates new value. All of open sources value is essentially squeezed, unless you're trying to retrain models that have already read all of our code and trained up on them and sell that back to us a second or third time.
[00:00:25] Introduction
Andrew: Hello, welcome to DevTools. FM. This is a podcast about developer tools and the people who make them. I'm Andrew. And this is my cohost, Justin.
[00:00:31] Meet Darcy Clark: Career Highlights
Justin: Hey everyone, we're really excited to have Darcy Clark joining us, Darcy You've done a lot of things in your career right now. You are working on this thing called volt or vlt Uh, do you say vault or do you say vlt
Darcy: I get people to say, the name twice, which is great. so however you want to say it, it's awesome. It's great branding, great marketing, VLT, VOLTS, whichever way you'd like to say it. we enjoy both forms.
Justin: Sweet well, uh Andrew and I are both lovers of javascript and of the ecosystem So we're interested to talk to you more about what you're building Before we dive into that though. Would you mind telling our listeners a little bit more about yourself?
Darcy: Sure. I'm the CEO of VOLTS, uh, CEO and founder. Um, formerly was the staff engineering manager at NPM, was acquired. by GitHub in, uh, 2020 and spent a good number of years there at GitHub, helped manage the GitHub CLI team there, as well as the NPM CLI team. Um, and historically have, uh, had a number of different jobs, did consulting for a number of years, but also helped lead engineering teams in some capacity, um, started a few companies in my early twenties as well.
Darcy: one specifically that's still around is a company called Themify. It was, uh, building commercial WordPress themes. So, uh, I used to sling some PHP back in my day, um, and built some theme frameworks and, uh, I've sort of stepped away from that ecosystem and, it seems like I did so at a good time given the current chaos that's happening in the WordPress ecosystem, but,
Andrew: Yeah, I've been a long time lover of JavaScript and a contributor to the open source ecosystem and foundations for a long time. so been slinging JS for, for 20 plus years. S
[00:02:15] The Birth of VLT: A New Package Manager
Andrew: o it seems like you have a lot of experience in package managers, but I'm sure the top question on everybody's mind is why do we need another one? What is VLT and what sets it apart from the pack?
Darcy: So I, I did the crazy thing, started a company, uh, based around some open source ideas I had, um, and decided to pull along, uh, my friends and former teammates, uh, Isaac Schluter, who I think has been on the podcast before the creator of NPM, um, I, I stole him away from. You know, his side projects at the time and got our team back together, pretty much, uh, at Roy Adorno, who was also, with me at NPM.
Darcy: And then GitHub also joined us, poached him from Google. And then Luke Karras as well, who was on our team. we pushed them directly from GitHub and brought him, uh, into Volt. and the reason why is really, uh, there's a few different reasons, but I think the big one is just most. Package managers are just clients.
Darcy: they are still, uh, sort of bound to the legacy infrastructure that is MPM and the legacy APIs that MPM provides to you. Uh, so there's not a lot of innovation happening on the backend or the infrastructure side, and so, um, I've spent a number of years focused on sort of products and engineering, uh, on the clients and, and saw all these opportunities that if we had.
Darcy: the right API is we could actually really, um, sort of, uh, push upstream some of the problems and, uh, I think the opportunities were sort of further upstream in terms of how we could provide new services for, for the ecosystem. So that's, that's the big rock that I think that we're pushing. Um, and I think it's the differentiator going forward is that, you know, we're, we'll be providing our own services and our own registry here, um, very shortly.
Justin: Package management in JavaScript is complicated, which I'm sure nobody knows that better than you. Uh, but I feel like it's only getting more complicated. Uh, so we have a proliferation of run times. So Dino and bun are two very prominent run times now, alongside node is sort of like server side run times.
Justin: Obviously, browsers continue to advance. We have E. S. M. so Now we have a new registry JSR. Um, you know, who knows if funds going to do their own registry or what that looks like. Um, and as, as the Dino team has sort of communicated compatibility in the ecosystem actually becomes really important. so if someone's using NPM, being able to install a package from JSR became like an important thing for them.
Justin: Um, And I, I assume that that's going to be a similar situation with what you're building, but you are explicitly trying to go, uh, like an NPM compatible route. So frame the, the context and the storytelling behind like what, not just like what the technical opportunity here is, but like, what is the real value that like having another package manager can unlock?
Justin: Or, you know, Maybe not even just package managers, maybe just the wrong terminology, but however you would explain it, like, what is the real value unlock that you're trying to push forward?
Darcy: Yeah, what I would say is like, we could use the term client. So what are the, the, for the client side, when we talk about clients, you know, NPM, PMPM, NPM is a lot of things, but the NPM client is sort of the de facto first party client to the service that is the NPM registry, public registry, and so, um, you know, yarn and you know, PMPM, and now, you know, you know, they all have, um, their own clients that essentially can communicate with MPM compatible registries.
Darcy: And what that means is there's this very poorly defined documentation around what the APIs are that would make you an MPM compatible registry. I screamed into a void for a number of years to try to get us to document this API because there's actually very few folks that That have done the work to go look at actually what the MPM first party client does, and then basically reverse engineer what those APIs and expected responses are.
Darcy: Um, so there's no like open API docs sitting somewhere that folks can use to be like, Hey, this is, this is what I can expect to, to, to either scaffold for myself to build a registry or that I can, uh, use to build a client. Um, and so I think that the. I guess you're sort of saying what, what's the opportunity here.
Darcy: And in terms of compatibility, we really have to look at like what the core primitive is that we're sharing today. And that is, you know, The defacto standard that NPM created, which is a package JSON really, um, uh, a package to NPM is just any, uh, tarball that has a package JSON in it. Um, it doesn't need a readme.
Darcy: It doesn't need any other contents. It just needs like a, a valid, uh, JSON file that's called package JSON in the root, um, and specifically there's two fields that need to be there, the name and version. That was not even the case, um, recently. Um, and we can get into this later, but, um, I, I surfaced, uh, a pretty serious bug in terms of the public registry, not validating the contents of those packages.
Darcy: So historically, you know, the NPM ecosystem in terms of the things that we're sharing, it's these packages. And so that, that sort of core primitive is the thing that I think is going to continue no matter what. And you see, even with JSR, they've, they've created, um, essentially, uh, an interface for anybody to consume a JSR package, um, Via sort of an NPM like service, um, that transforms the projects into packages, like so a project that has a package Jason, whether or not you're using just the Dino Jason config, um, they'll, they'll sort of mutate or transform the project.
Darcy: So it can be consumed by. You know, these other clients, um, so there's some compatibility that they've baked into their registry. And I think that's going to have to continue. Uh, we've actually done the same thing with VSR. We have a pretty, um, like we are basically giving you an NPM compatible, registry.
[00:08:43] Innovative Features of VLT
Andrew: So focusing on the client a little more, as Justin was saying, JavaScript's gotten pretty complicated. Uh, it might not just be one package and one repo. Uh, typically we work with monorepos. So does, uh, does volt help me build monorepos? Is it just focused more on like a subset of use cases? Like, what are you guys aiming for here?
Darcy: Yeah, so by default we have workspace support or monorepo support. Um, and actually we do with a few different capabilities inside of the Vault client. We do support, other, Package managed, project types. So PMPM, uh, you can be using a PMPM to manage your dependencies and you can use the Vault GUI to actually explore your, your project, which we, we thought was, interesting use case.
Darcy: And so yeah, workspaces are definitely, uh, first class, uh, in, in terms of how we wanna support them. Um, we have taken a similar approach to, to how PMPM, basically took a importer, uh, type strategy. MPM, when I was there, we actually implemented MPM workspaces and V seven. it was one of those things that was a long, outstanding.
Darcy: Need from, uh, you know, 2016 when, when yarn sort of introduced workspaces, we, it took us a long time to get there. but that was sort of tacked on to NPM. And so you could really tell the APIs weren't native to, to the client. Um, and we're taking a bit more of a holistic strategy with how we look at not just workspaces, but dependency, um, uh, traversing, you know, Um, you know, all together and, uh, that starts with having sort of a holistic understanding of your dependency graph and your projects.
Darcy: and the way that we're doing that is by treating your dependency graph, uh, just like you would the DOM. So instead of, uh, you know, in the front end, you have the document object model, which is created by browser HTML markup while we, you know, I believe, you know, there's something out there like your dependency object model, where you can essentially, um, traverse all your dependencies in a standardized way, um, or at least the way that we believe can be standardized and use a common language to essentially traverse, um, that graph.
Darcy: And so we've created a, uh, dependency selector syntax that essentially mimics what CSS looks like to the DOM. And essentially having. Primitives on the nodes, like the type, let's say like a dependency is a workspace or it's not a workspace, um, makes it really easy for you to write a selector essentially that allows you to filter, um, your dependencies for the purpose of running tasks or doing installations.
Darcy: and so instead of having sort of tacked on flags for all your sub commands, uh, to run, No scripts or to install, you know, dependencies into one, one project or another, you get sort of a common language that can be used across the entire tool.
Andrew: So could like, uh, So for a concrete example, like in our monorepo, uh, we have a bunch of scripts that we like to run in CI. Uh, but installing the dependencies for those scripts kind of requires a lot of work. If you want to do it the right way, what we ended up doing is throwing them in a workspace and then filtering the install by, uh, saying just install the dependencies in that workspace.
Andrew: That's a lot of work. It is not a fun, fun way to code. Does this query syntax kind of help me not do that and like rely on the queries more so?
Darcy: Totally. that's the idea. Um, we're, we're coming up with like a broad set of, um, I say broad, there's essentially like three types of queries that are supported by pretty much every command. And it's the context in which you're executing the command, whether it's install, it's run, it's exec, those are all supported by Vault today.
Darcy: or whether it's the target. Um, so this is, it's very similar to, if you imagine, and I like to always leverage like the webs. Web platform. It's, it's like essentially event handling, right? So when you're considering like, um, what's the target, what's the, uh, what's the context or the scope, um, and these two things can be set for every command.
Darcy: And you, I can imagine you could set using query syntax, um, a very, uh, broad, set of, of, Dependencies. And the great thing about this is that it's super expressive. It's just like CSS. Although I know many folks have. Moved to, to tailwind and for have forgotten how to write a CSS selector. but, uh, yeah, you can essentially just write a CSS selector, um, utilizing all this metadata by your dependencies.
Darcy: And you can do something, uh, that I don't think is possible with. Just the, uh, sort of legacy arguments for like prod versus dev versus, you know, optional dependency, flagging and filtering. I will give a shout out to PMPM's, uh, filter flag. If you've ever used it in projects. I know a lot of folks use a PMPM for monorepos.
Darcy: I think they've done a pretty good job with that, that filter, but I still think it actually suffers from a lot of the limitations of not having, you know, a. Actual language query language.
Justin: This is, this is like pretty, pretty interesting. I think there's a lot of tooling out in the ecosystem that re implements a package resolution or whatever. Um, I mean, I've, I know that I've used an NPM module before that was just written by someone random that like kind of does this, it's like figure out like how node would resolve something or whatever, and like, you know, try Slap together script to find versions or whatever.
Justin: So I like that. I like this, uh query command for sure That's very interesting for like maintenance tasks um, maybe that like Plays into another thing. I noticed that you have a mermaid command, which is cool for visualization so if you like want to write some rule and like see the visualization of like, okay here like Maybe the direct dependencies or the, the, uh, transitive dependencies of this, like one package or something.
Justin: And it'd be like really interesting way to view that.
[00:14:57] Ad
Andrew: We'd like to stop and thank our sponsor for this week, but we don't have one. So if you'd like to sponsor DevTools FM, head over to DevToolsFM slash sponsor to apply. And if you want to find another way to support the podcast, head over to shop. devtools. fm, where you can buy some merch and rep the podcast.
Andrew: With that though, let's get back to the episode.
[00:15:16] Exploring the VLT GUI
Justin: Um, but you have a, you have a GUI that you're showing some of the stuff in and giving more capabilities for. Um, and I, and I think that that GUI is like a, sort of a next level of like a vision of what this client could be or what a holistic client could be.
Justin: So, what are your goals there? What are you trying to do? And what does it do today?
Darcy: Yeah, so we we kind of Went back to first principles in terms of what we wanted ourselves. I know I consider myself to be a full stack developer. I used to love to, you know, I aspired to be a unicorn in that I, you know, it was doing front end as well as backend. I've always been really jealous of the developer tools experience we have in the browser.
Darcy: Um, GUI experience for us was a way to try to unlock what I felt like we had been hiding from so many developers, at least in the past. In the server side JS, uh, realm where, um, I'm sure you've seen the meme, you know, the, the, the most massive thing in the universe is your node modules, like this black hole.
Darcy: And then it's your node modules folder that always made me sad, you know, as, as somebody who was like leading the charge with NPM and, um, I turned around the ship a bit there. We had gained, you know, better adoption. I always felt we did a disservice to the ecosystem by not allowing developers to feel, uh, inquisitive and learn more about what was in node modules.
Darcy: It should not be a black box essentially. Um, and so the GUI is really a mechanism for you to explore and actually see why something is, is installed. It's also a visual way of traversing your dependency graph that isn't, um, Some sort of like very large, uh, you know, octopus looking, you know, dependency graph.
Darcy: It's actually, uh, it's a interesting sort of, um, three column, uh, traversal, um, for, uh, nodes, like edges in and edges, out for, for nodes, um, in that graph. So we actually think it's really a unique way to, to visualize, notes in your project or your dependency graph. And so you can see your dependencies on the right hand side, the currently selected dependency or the list of dependencies that are selected and then any, uh, ancestry.
Darcy: so you can quickly click around and navigate. The really interesting part here is that as you're clicking around and using the GUI is that we update the query selector. Um, so. Uh, if you're not very familiar with CSS, let's say, um, you are basically given a breadcrumb, for how, you know, to essentially find that exact same query again, and you can share that with your team.
Darcy: You can essentially use it again in, uh, you know, running tasks or, um, you know, using it to filter, um, projects for any, uh, any kind of reason we look at it as sort of like the first step towards like making it super easy for you to build reusable sort of queries in the future. Uh, and we're doing some work right now to, to make that experience even better.
Darcy: but yeah, we think the GUIs is just the thing. The getting started in terms of the opportunities of, uh, essentially appending even more and more metadata to each one of the views that you see for your, your dependencies. Um, so we can pull in information from socket. Um, I'm sure you know, those folks, um, or other kinds of information from, you know, the third party registries like downloads counts, et cetera.
Darcy: Um, so we imagine that to get more and more, um, uh, Uh, be like more of a rich experience than you see today with like the NPM registries, uh, public facing website.
Andrew: Yeah, I was playing around with it. And like immediately I found the use for it. Cause I was just like installing random dependencies to see what the lock file looked like, and just kind of like gauge what was happening. And I accidentally installed next dot JS instead of next JS. And because of the tool, it was like immediately apparent that, Oh, this is like some random library that's.
Andrew: Somebody put out that has no, no relation to what I was actually wanting to install. So like, I can see like, there's so many layers that you could add on top of that to make that experience even more awesome.
Darcy: Yeah, we think we're just getting started with it. Like we're, we're super excited about what we'll be rolling out. Like there's so many, um, what we consider to be like, there's a thousand paper cuts. Um, my, I know my good friend Wes Boss was yelling about not having, you know, sorting by downloads counts for on the MPM public registry, things like that are, are essentially just pieces of metadata that we can, again, append to You know, uh, you know, nodes that then we can give you a way to in a selector, uh, a reasonable sort of, um, mechanism for sorting by or finding, let's say, Dependencies that have greater than or less than a certain number of downloads potentially.
Darcy: Right. So we've added a bunch of pseudo classes and pseudo selectors essentially that are bespoke to, you know, our use cases. Um, uh, but we imagine there's even more like, you know, download counts or, um, in the case of, you know, let's say a typo squat. Um, You know, we can pull in information from third parties that are willing to integrate with us, including someone like socket, um, to, to grab that kind of metadata and then provide you with again, a way to filter or sort or, or search with that.
Darcy: So, uh, really laying the foundation with query selector syntax and the dependency graph. Um, we can just provide a great visual experience that, that plays with those, uh, tools under the hood.
Justin: A good use case for this, it makes me think back. So we had, uh, Jordan Harband on and he, he was sort of making the case that it's like, not about the number of packages that you have, uh, as dependencies, but it's about the number of people who like maintain those packages that like chain of trust. It's like how many people are involved and like, You know, it wouldn't be necessarily a trivial thing to just like, Oh, like query all my packages and transitive dependencies and tell me like, what is the complete set of like authors or, you know, of these packages.
Justin: And, um, I feel like something like this could, could do that. And especially if you're, you know, down the line, if you do a lot of enriching information, like get up metadata or something like that, that you can pull in and integrate with it too. It's like, that can be really, it could be really compelling.
Darcy: Yeah, I think the use case you just brought up there for discovering authors and contributors to packages, that kind of metadata, you hear that different use case in many different shapes and forms. Like there was, I think, five or six different commands in the NPM CLI that all did. The same thing they were essentially just taking different slices of the pie, like different, um, a use case for finding all my outdated dependencies or, or the use case of, you know, listing all my vulnerable dependencies.
Darcy: Right. So, um, you know, people came to us asking, can I list all the licenses of all my dependencies and all of these are just different, um, sort of slices of, of doing a query and having a different view or shape, um, of, of the, uh, The output of that, that data, um, and so by providing really like expressive, um, and, and I think, um, fundamental, uh, new language, I think we can unlock a lot of.
Darcy: You know, a lot, a lot of use cases we haven't even thought about. So like just providing a way for you to get that metadata or search by that metadata, um, is, is I think going to unlock things for, for the ecosystem. so we aren't the blocker anymore. You're not waiting for somebody to implement a new flag or a new command to, to get, you know, uh, all the authors, list all the authors of all their dependencies.
Andrew: So something that docs calls out a few different times is secure.
[00:23:14] Security and VSR Registry
Andrew: So what does secure mean in the context of the VLT client?
Darcy: So security in the context of the client is not as important as security in the context of the registry and the infrastructure. And so, um, clients are beholden to the network and and what you'll find is, um, and I experienced this firsthand at GitHub. Um, there's so many times that people want to file. Uh, issues against the clients that I managed both the gap CLI and the NPM CLI folks constantly tried to tell us that we, we had, uh, you know, bugs or security issues with the clients.
Darcy: but there's inherently a trust model that basically says if your network is compromised, like, you know, you, your client is going to be interacting with a compromised network. Um, and so. There are integrity checks baked into since, you know, 2016, 2017, lock files have existed. and there's integrity checks baked into pretty much all our clients.
Darcy: That's, that's the mechanism we have today in terms of storing and validating the package you expect, um, to be downloaded is the one you, you downloaded. but we don't do anything beyond that. many clients, uh, you know, Basically you're beholden to the configured registry. Um, in terms of security where we're maybe deviating from what the MPM public registry does and what some other, uh, you know, um, enterprise registry, uh, providers have been doing is we're actually doing a manifest validation and the package, uh, to, to prevent, uh, what's called manifest confusion, uh, validating that actually the manifest and the integrity.
Darcy: Um, or, or the manifest and the, the, the package JSON that was shipped in the tarball, uh, has the same, same values, same set of values and keys. Um, we basically throw out the manifest because it can be essentially, um, uh, manipulated during a publish. Um, and so those two things can be not the same. And so you end up with essentially, uh, corrupted caches or, uh, the clients don't know which to essentially use when you're deciding to install dependencies or run scripts for, for a package.
Darcy: Um, so the source of truth should be the thing that was actually signed. Um, And that is the package Starball. Um, so if there's any other client authors out there or package managers, authors out there that are listening, like, please, I implore you to throw out the manifest information. and, uh, and pull extract the actual contents from packages. So that's, that's, that's security, I guess.
Justin: this is a good lead in to VSR, which is the registry service. You know, you kind of like talking about this with security implications and thinking about like how security is handled a lot on the registry side. So, uh, we'll dive in and talk about some more features, but why don't you give us like a, like an overview of VSR and maybe how it compares to some of the other, like NPM compatible registries.
Darcy: Yeah, so VSR, we looked at. What was out there and there's, uh, there's a few enterprise offerings, Nexus, um, Sontypes, Nexus product and JFrogs artifactory. Um, the, the price points. Yeah. I see some shaking heads. If you've ever used these, um, you'll know, uh, both how clunky they can be. Um, some of the actual.
Darcy: You know, implementation details can, can muddy the waters for folks. You can essentially do something which allows you to proxy or intercept, and, conflate, um, package packages, which I think is, is again, almost like a bad practice or potential for security issues for teams. And, and you see this with dependency confusion attacks and, and, and things like that.
Darcy: And so, um, at least with providing our own, you know, registry software. We decided that we want to open source it. Um, or, or I should not say open source. We want to fair source, um, a product. So the VSR, um, code is essentially a fair sourced, uh, licensed, MPM compatible registry that you could host yourself if you, if you want to for private packages or for your team.
Darcy: Um, and it's. serverless by default. So the overhead of actually running it is like super minimal. That was another, uh, something that I heard time and time again, was how painstaking it was to spin up, um, the only other two alternatives that I know of in this space. Um, it's, uh, it is pretty good. Sort of competitive with, uh, Verdaccio, if you know, is the only other, I think, true, uh, sort of open source or free registry proxy or self hosting tool.
Darcy: Um, and yeah, I think there's a lot of opportunities with being, you know, having it out there and available to the ecosystem to, to both. Contribute to collaborate on, um, I think it shines light on what we're doing internally on in the code. So, um, again, ideally, that's more secure, but we're also going to, you know, work closely with the ecosystem in terms of how we develop it.
Darcy: So, um, the opportunity here is to improve and adapt the, um, Endpoints and APIs that are actually in the infrastructure, meaning that we can build something that, uh, you know, doesn't stagnate like the MPM public registry sort of has for, for the last decade or so. So opportunity to both, you know, build something that's more secure, ideally at a price point, that's way more competitive that you can imagine those enterprise, registry clients or registries are.
Darcy: And, um, yeah, also improvements to the, the clients that interface with it.
Andrew: Yeah, I had the opportunity to work with Artifactory a few years ago and not a fun experience. We like to publish packages a lot and at some point their, their registry just stopped working and they're like, Oh, you, you published too many packages. You need to start deleting some. Uh, yeah, that, that was the look on our faces too.
Andrew: We were very surprised. So it's nice to know that, uh, somebody is thinking about making a good version of this. And it happens to be from the people who make NPM. So it's a little more trustworthy that they're going to shake things out, right?
Darcy: Yeah, I thought it was the biggest miss. Um, I mentioned that there was a lack of documentation for the registry. I thought it was the biggest miss that people had to, um, like, uh, the Verdaccio project is perfect example of, of, uh, folks really. Having to work hard to try to reverse engineer, uh, what it means to self host and, an NPM instance or NPM compatible instance.
Darcy: And so, uh, I don't want to have that happen again. So we're, we're starting out, uh, with, uh, you know, you know, this for fair source model that allows everybody to see exactly, uh, what we're building and can, uh, you know, As long as you're not competitive, there's no competitive use, then you can essentially use and host one of the instances yourself.
Darcy: This is a similar model to Sentry. Sentry sort of has coined the first source license and we're using that for VSR today. We also do provide a managed Uh, version of, of the VSR, uh, instance, and have a few different use, like primary use cases, both, uh, for folks that are selling their own software, but also for folks that want to move off of either MPM private packages, which were actually cheaper than, uh, as well as, uh, you know, folks that are using, you know, artifactory or Nexus.
Justin: Yeah, it makes a lot of sense. I mean, I think there is a lot of opportunity in this, uh, this space. Um, so, you know, it's like things that you list out in your docs, but specifically, like, if you have a team that's wanting to use a private registry as a distribution channel, a lot of the, Tools out there are not built for that.
Justin: They're not designed for third party access to a private registry. It's, it's assumed either public or private. And you have folks who go through this process of, yeah, I don't know, just trying to work that out, um, and doing a lot of weird things. And so it is, that's an interesting model. And of course there's like all the things that come with enterprise, like compliance, um, and.
Justin: Observability and all those other like important money making things that like, uh, if you, you know, are a sock to company and like at a certain scale that you just kind of like, have to have these things. So, um. I think there are a lot of like obvious wins. Um, and you know, this is just a side commentary.
Justin: Um, I just want to say like, I appreciate that y'all are doing this and I appreciate that y'all are doing this in the same way that I appreciate that Dino and Bun exists because they have put a positive pressure on Node to improve. And I hope that you have the same impact, uh, because it's like, regardless, I was good.
Justin: It's like heavily used and we just need better package infrastructure across the board Um, I mean regardless, you know, hopefully this like lights a fire and there's somebody at github They'll be like, okay, maybe we can do better I hope
Justin: Or maybe you'll be the new npm
Darcy: Yeah, the old new NPM. we're, we're definitely still close with the team that is there and then the folks that are still left there. and we, um, as you say, do hope it's a positive pressure. Um, there's just a number of things that I think are out of scope for that team and even GitHub itself to, to sort of support, and, uh, you know, I like to say, oftentimes we want to be the Google of packages and, um, today get both get hub and MPM don't index.
Darcy: you know, dependencies or packages that aren't on their registries. Um, so that's one, you know, sort of, if you're talking about security, if you're talking about, um, observability, um, and if you're talking about, you know, uh, potential for, for, you know, uh, money making opportunities for, for those enterprises, like, you know, you want to know whether or not you're about to install a package from like a third party, you know, uh, somebody's private website.
Darcy: Um, and today you can totally do that and distribute a package on the npm registry that actually references all these different sort of bespoke, uh, package, uh, hosts. Um, and there's nothing preventing the clients. The various clients from, from downloading and essentially, um, opening you up to, to all these different, you know, different origins that you didn't expect would be a part of, you know, your transitive dependency stack.
Andrew: Yeah, the, the homepage for VSR calls that out. So there are, there's tools built into it that like prevent that from happening. Like say, like depending on a GitHub repository where somebody could just like switch the code out unbeknownst to you.
Darcy: So there's, there's two ways we're looking to address that. One is, uh, indexing. fully, uh, and fully realizing your dependency graph, uh, during the indexing or publishing of a package into, into VSR, or in the future where we will actually have our own hosted infrastructure for this, uh, we can be treated as sort of your new upstream.
Darcy: So we could be a layer in between you and the NPM registry. Um, So today, um, NPM is, is pretty, the, the infrastructure is pretty dumb. It essentially sits there and just hosts the static references and has sort of these weak references to your dependencies. And you'll actually see this in, in practice. If you go to a package page today and.
Darcy: You can go find a number of examples of this where it will show the number of dependencies in the tab for dependencies, and it's completely inaccurate or the name that it's referencing is not the actual package. It will link you to, you know, a react in the registry, even though that react that that dependency is referencing is a.
Darcy: GitHub fork or a repo fork, um, or a private package tarball that again, somebody has potentially patched, you know, that, that react. And so the, you know, conflating the, that package name react with something in the registry and something that is going to come from anywhere else on the internet is extremely dangerous.
Darcy: And so what we will be doing first and foremost is treating those. Separately, and essentially when you would come to the experience to view packages in VSR, you would be able to essentially see that these two things are different. And so by indexing them independently and creating a strong relationship between those, those packages, we will be able to give you a more secure essentially experience.
Darcy: I was just going to say this all goes back to the value of enriching the dependency graph and like making that a first class thing. Uh, I feel like there is like actually tremendous value just like right there, you know, like not doing anything else. I mean, obviously there's a lot of. technical opportunity and product opportunity across the board. But, um, you know, when it comes to like, I think this is true for the whole ecosystem. We just have to acknowledge that like there's an ecosystem outside of just our bubble, right? This is like, if you're publishing a package to NPM, um, there's still could be dependencies and JSR or GitHub or whatever, you know, someone puts out there.
Justin: And, uh, I think. If y'all are able to sort of like coalesce this, like really rich enhanced graph of like, okay, here are all the possible data sources. And here's the metadata from the sources that we like originally provide you. Then that's, yeah, it was going to be tremendous.
Darcy: Yeah, we, we think there's a huge opportunity once we have those strong relationships. it means you can do some really cool things, um, including, um, instantaneous lock file generation, um, given that you've indexed the entire ecosystem or, or known, let's say NPM package ecosystem and resolved all those dependencies, um, because you know, you essentially have a service that's providing that, um, you know, that, that offering, we essentially know about all the, the packages available to you.
Darcy: So you don't have to do any of that, you know, range, you know, December range calculation on the client and waste a lot of compute. you can essentially pin or distribute a lock file immediately. Um, so. Yeah, there's some, some opportunities there to also provide, you know, performance benefits, not just for the compatible client, but even manipulating the, um, or, or modifying the hosted manifests, um, in the APIs and pinning, those can avoid a whole bunch of computation for, for the legacy clients, even so, you know, we could even potentially for speed up fun and Dino.
Andrew: So, so you said that you guys want to be the Google of JavaScript packages. And I find that to be a very interesting statement that I want to unpack. Cause like right now, uh, it used to be that there was just MPM. You had one place to search, but, uh, things like JSR have come up where it's a completely separate registry.
Andrew: They do very different things. Things than what MPM is doing. Do you guys plan to like, kind of have your own public registry that is like melding all of these and making it into one cohesive experience?
Darcy: Yeah. So phase one of, of this has been introducing the client, which, uh, essentially, um, interacts with MPM and any other MPM compatible package, uh, registry. phase two was introducing a VSR, which we think is, uh, sort of unlocks the ability to self host and gives you some freedom to distribute packages yourself.
Darcy: Um, again, um, aimed at a couple of different use cases. We heard time and time again from folks who were either selling their own software or had, you know, private packages that, you know, and teams that they just can't imagine paying, you know, the 30, enterprise licenses and, and getting a fraction of the, uh, the real APIs or tooling, um, to, to support just private, uh, software.
Darcy: software hosting. so that was sort of phase two. Phase three is to introduce a, uh, an interface between those public registries, essentially indexing NPM, uh, fully and fully resolving, essentially doing a spider's walk of, of the fancy graph for you. Um, and JSR is on there as well. So the opportunity there would be to.
Darcy: Uh, point to, uh, us as an upstream for either your VSR, um, uh, registry itself, or directly with your client point at our, our registry in the future. I do not think we'll probably introduce our own, uh, publishing destination, from the get go, but it would be on our radar to eventually have you, uh, be able to, uh, publish into our registry as well.
Darcy: I think the opportunity here is to, to not disrupt the current, uh, ecosystems way of distributing. If, if folks continue to like to publish the NPM, that's totally fine. Um, I think there's a lot of benefits to using us to consume from that ecosystem. Um, And, uh, yeah, if you ever are wondering where the VLT acronym comes from, it's actually virtualized logical trees.
Darcy: So the whole idea is that we're actually trying to, uh, virtualize and, and, and, uh, push upstream the resolution, which has so often, you know, being wait, you know, this redundant compute that we all do on our, our systems and our CI so often, and, and sort of wasting, you Many cycles, uh, resolving the same dependency graphs.
Darcy: And if we push that upstream and allow a service to do that for us, we can get a lot of benefits.
Darcy: Is that
Darcy: the answer to the Google indexing? It's a big, it's a very broad, you know, what does Google of packages look like? Uh, I think for the JS ecosystem specifically, it's, it's let's know what we're about to install, uh, and not have it be a surprise for you.
Justin: I think just like, just like the graph, the dependency graph, being able to like bring that front and center and get enriched information about it. I do think that this vision of just like, you know, like, You know what? There's a lot of package registers out there, but like, here's where you go to find it.
Justin: Uh, You know, because that becomes like pretty big, right? That's like, uh, a challenge in the space now is like now more than ever before. I have so many places I have to go to look for something. It's like looking on NPM and one interesting challenge that I'm really curious how y'all deal with is all the spam that like NPM gets hit with these days, which is, you know, I, uh, I have a lot of.
Justin: Sympathy for the team having to deal with that, um, the, uh, like tea, like packages or whatever problem. Um, but. Anyways, like between, you know, actually searching on NPM and finding results or like going to JSR or, you know, just searching GitHub, you know, it's like, that's a, that's a real challenge, you know, sometimes to find where are the high quality packages and, you know, actually how can I find it from all the noise?
Justin: So just a integrated search product where you can install from one place. It would be super valuable, you know.
Darcy: Yeah, I think the missing, there's a lot of missing, like I said, a thousand paper cut features that I think that we can execute on the. Real problem I think NPM had was we stuck too long to the, you know, PQM scores that were created by NPS. I think NPS. io, you know, we forked their indexing and their, uh, quality, uh, metrics back in the day.
Darcy: And, and NPM didn't move off of that until I think. Just about a week ago, they finally switched to just allow you to sort by downloads. And I think really, if you're talking about trying to address even spam in registries, or finding high quality indicators for good quality or high quality packages and during that discovery phase.
Darcy: I think you need more than even downloads. You need, you need a, you know, rich, uh, set of, set of, uh, metadata associated with those, those packages. Um, and so the opportunity here is to partner with folks that are also doing this work. Uh, I've referenced a couple of times already, but you know, socket. dev I think have been doing an amazing job at doing package analysis, whereas there's a lot of others.
Darcy: Uh, you know, security companies have focused on the CVE ecosystem, which can be, uh, hit or miss in terms of the quality, um, actually doing, you know, stat code analysis. And sometimes, uh, I think they're even potentially doing active or runtime analysis to, um, to actually discover, you know, key insights about the quality of a package and potential malware that might be that in there.
Darcy: I think that's way. More useful than, uh, you know, alerts or advisories, you know, uh, post, post installation, obviously.
Andrew: before we move on to the future facing questions, I want to ask you a little more about hosting paid packages. So it's another thing called out on the homepage, but that seems really interesting to me where you might be this like Google in this first stop before I find packages. If I could also find packages to pay for, that would be really interesting.
Andrew: So is that something you guys are working towards also?
Darcy: Yeah. So when I got to NPM, it was a, it's a bit of a firefighter moment. It was going through a transitionary period. Uh, I think it was 2019. Um, so I was hired about a year before we were, eventually acquired. Um, and one of the key things that folks were talking about at the time was both spam and their terminal.
Darcy: And we, we could quickly address that. but at the time actually for us was running his funding, um, Prompts and ads in your, your install output. Um, if you remember that, and that got a lot of noise in the ecosystem at the time. and one of the first things that we shipped in the CLI was the funding feature, um, the idea that you could, uh, you know, essentially.
Darcy: codify some metadata about where to find, you know, um, sponsorship or funding opportunities for the packages that you're consuming. Um, and GitHub did this as well and introduced something similar around the same time, uh, with sponsorships and actually activated, you know, a mechanism to, to do Stripe sort of donations and, and monetization, um, that way.
Darcy: Um, but donations are not, you know, At all a mechanism for, for sustainability. Um, and I've, you know, I've seen this play out many times, uh, marketing budgets go get slashed the minute times get tough and there goes the donations. And, uh, you know, the developers no longer can sustain sort of the open source model.
Darcy: And so many people came to us and said, Hey, like, I would love to, License, you know, I have a separate license for this software. How can I like restrict it? Um, and so we've actually got two. customers right now with two different use cases, sort of the private packages and another one that is essentially a SAS company itself that sells their, their software and is hosting, uh, packages using essentially VSR.
Darcy: Um, and they have roughly 5, 000 users. Customers themselves. Um, so yeah, the future for that definitely looks like, you know, we, we could help, um, developers essentially get paid for or, or license their packages. Um, but what is more interesting to me is actually finding a way to, um, operationalize open source in a new way that.
Darcy: Um, creates new value, um, because all, all of open sources value is essentially squeezed, there's no, no more, no more money or value to sort of get, get from it, um, unless you're trying to retrain models that have already read all of our code and, and trained up on them and sell that back to us a second or third time.
Darcy: Um, but, uh, I think there is an opportunity for, runtime and a new runtime. Our compute offering essentially, uh, that operationalize packages essentially as a service. and there's a few, there's a whole category of packages in the ecosystem that, uh, lend themselves really well to this. Um, you can imagine like Moment.
Darcy: js. Being used as like an API, um, uh, consuming that and essentially paying us for the compute, um, offloads, a couple of things, it flattens your dependency stack. Uh, and then this service essentially would rep share the money back to the ecosystem. Um, so all of the dependencies in that stack in that execution would essentially get.
Darcy: Uh, money. So in terms of getting developers paid, the long road for us will be this sort of, uh, fourth or fifth phase of the, the platform to, to ensure that the ecosystem actually becomes sustainable long term, um, and there's benefits on both sides in the way that I see, you know, us going forward with that product is the benefit for the consumer is a flattened dependency stack.
Darcy: You know, less DevOps, uh, and they also get to contribute, you know, into the ecosystem, um, and the open source ecosystem gets to benefit from rev sharing, you know, uh, that product from that product. Um, so creating new value and then, you know, encouraging folks to, to love or adopt or, or tell their, you know, uh, teams to, to get us into their, you know, uh, CI pipelines, uh, You know, by paying, essentially paying back, writing the check that you're not getting from get hub today, writing that check for, for you as a, uh, author or maintainer of a dependency, you know, deeply nested potentially in the transit of dependency stack.
Darcy: So, uh, while there's opportunities to do what you're saying in terms of market, a marketplace play, I think the longterm sort of sustainability is where I think we're going to invest time, uh, probably towards the end of, of this year.
Justin: Well, that's exciting. I hope you're successful. I I've heard similar ideas and different flavors. And I think it's still something that's needed in the space. It's like one of the sort of classic problems in open source that we've talked about on this podcast, especially time and time and time again, is you have these incredibly brilliant engineers. Tons of time and to open source projects that companies derive a ton of value out of, and then, you know, struggle to pay their rent or whatever. I don't know. It's like, it is a weird, it's a weird ecosystem, but anyway, I hope that, I hope that whatever you find out there is, is good for everybody involved.
Justin: Um, with that, let's.
[00:51:02] Future of JavaScript Package Management
Justin: Talk about the future. Uh, one of the things that we like to ask is in a wrap up question is a future facing question. And for you, it's pretty obvious. What do you think the future of JavaScript package management is going to be like?
Darcy: package management, and I think package or dependency management. Um, The focus has been performance for the last couple of years. I don't think that that's going to change. Um, things are going to get faster, uh, constantly. Bun and Dino obviously have really doubled down on, on that. Um, and the runtimes are always getting faster.
Darcy: And I think that that's important. I had a number of customer calls in the last. The number one line item for most people that I talked to was, uh, CI, like, uh, give actions was the number one, uh, cost, to the developer. Tools and, um, and pretty much nobody else had any other costs, you know, like nobody's paying for anything else.
Darcy: but, but CI and compute, um, so performance, if you can save somebody time, uh, you're saving the money. And so I think that's going to continue to be the, the future of, of our ecosystem is folks finding ways and opportunities to, to, to get better. Increased performance. Um, the second thing that I would say in terms of what I see as the future is just freedom and choice.
Darcy: I think, um, although we have a lot of that in the ecosystem already, uh, at least in package management, it's not, it's not that, uh, saturated. I would, I would say, uh, you know, some folks feel like there's a fragmentation of having five or six package managers, um, we'll look at, you know, Look at, uh, you know, test, test runners, or like, look at other kinds of, uh, shapes of, of libraries and projects.
Darcy: Like there's a lot more, um, competition in, in other places in the JavaScript ecosystem. So I think that the opportunity to, to, you know, offer a new alternative to, especially as I think the, the biggest, um, Player, which is NPM an alternative to NPM is just more freedom, more choice. and I think, you know, I I'm a big fan of what JSR is doing as well.
Darcy: And I'm a big fan of, uh, Dino and those folks. So, um, I'm all for more competition and ideally we, we get better, uh, tools, better, you know, results as an ecosystem because of that. Um, and then I think also we're going to get a bit more secure, um, and not security forced to us from like CISA and, and S bombs that are completely, uh, for, you know, not, not usable at all and not, not actually tracking, um, real, real goods or, or, um, You know, contents of software, but I think we'll get real, uh, security solutions like, socket or in our case, what we're doing is, is just, uh, you know, truly making a real impact by, by doing and making logical decisions about how we deliver packages to you.
Darcy: Um, there's just a ton of low hanging fruit in this space. and it doesn't, doesn't require any kind of blockchain, you know, technology or anything like that.
[00:54:20] Conclusion and Final Thoughts
Andrew: Well, that wraps it up for our questions. I'm excited for the vision you've laid out. You've already built a lot of cool things and you're planning on building some even cooler things. So I wish you luck because I think the JavaScript ecosystem needs it. And thank you for coming on.
Darcy: Thanks guys.
Justin: Yeah. Thanks Darcy. This is, this is incredible. Excited to see what y'all build.
Darcy: Cheers.