You are watching news of Coinbase. Go back to all news.
-
View all news of Coinbase
Posted on July 1, 2020Coinbase Custody selected by 21Shares for Bitcoin ETP
Today we’re proud to announce that 21Shares AG has chosen Coinbase Custody to store its digital assets for its Bitcoin ETP. 21Shares AG is listing its 21Shares Bitcoin ETP (ISIN: CH0454664001 — WKN A2T64E) on the Deutsche Börse’s Xetra electronic venue on Thursday July 2nd 2020. Coinbase Custody will serve as custodian for the underlying assets of their Bitcoin ETP.
21Shares selected Coinbase Custody after reviewing our institutional-grade offline storage solution, which includes world-class security, regulatory compliance and insurance coverage. Coinbase is committed to serving a wide spectrum of institutional clients and we are excited to be the chosen custodian for the 21Shares Bitcoin ETP, Europe’s first physically-backed bitcoin ETP.
Hany Rashwan, CEO at 21Shares AG, says, “21Shares is delighted to bring this historic product to Europe. We are honored to give investors access to the best performing asset class of the last decade and we believe the next ten years as well, through our simple, transparent, and regulated crypto ETP product suite.”
Coinbase Custody is the world’s largest and most trusted crypto custodian. To learn more about Coinbase Custody click here.
Coinbase Custody selected by 21Shares for Bitcoin ETP was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
-
View all news of Coinbase
Posted on June 25, 2020Compound (COMP) is now available on Coinbase Earn
Earn COMP while learning how to borrow crypto and earn interest with Compound
Coinbase’s mission is to create an open financial system, where anyone in the world can participate on equal terms from their computer or smartphone. To bring this vision to life, we’ll need to make blockchain technology more accessible, both in the sense of making cryptocurrencies easier to obtain and easier to understand.
Coinbase Earn is a trusted source where customers all around the world can educate themselves about new developments in crypto and earn assets as rewards. Starting today, Coinbase customers can start earning Compound (COMP) by watching lessons and completing quizzes about the Compound protocol and its governance token COMP.
Compound (COMP) is an Ethereum token that governs the autonomous Compound protocol. The protocol allows anyone to borrow and lend Ethereum tokens through a decentralized market. Lenders earn interest on the crypto they supply to the protocol and borrowers pay interest to borrow it.
The ability to earn crypto assets has become an increasingly important function in the crypto ecosystem — alongside buying, staking, voting, and mining — especially when paired with education.
Since the launch of Coinbase Earn in 2018, we’ve announced nine different assets, including 0x, Basic Attention Token, Zcash, Stellar Lumens, EOS, Dai, Tezos, Orchid and now Compound. In less than a year, asset issuers offered over $100M in crypto to distribute to our customers, and we have launched Earn internationally. As we expand globally, millions of people are gaining access to a trusted, secure, and legal bridge to the crypto economy — even if they don’t have a credit card or bank account with which to buy crypto.
Check out all Coinbase Earn campaigns through our homepage or access them directly through the Coinbase mobile app.
You can sign up for a Coinbase account here and download the iOS or Android app to start earning crypto today.
Coinbase receives a servicing fee from the participating asset issuer. Consideration for Coinbase Earn is wholly independent of our digital asset framework for new listings. Click here for our Earn FAQ and terms.
Compound (COMP) is now available on Coinbase Earn was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
-
View all news of Coinbase
Posted on June 25, 2020Compound (COMP) is now available on Coinbase
Starting today, Coinbase supports Compound (COMP) at Coinbase.com and in the Coinbase Android and iOS apps. Coinbase customers can now buy, sell, convert, send, receive, or store COMP. COMP will be available for customers in all Coinbase-supported regions, with the exception of New York state.
Compound (COMP)
Compound (COMP) is an Ethereum token that governs the Compound protocol. The protocol allows anyone to supply or borrow Ethereum tokens through a decentralized market. Suppliers earn interest on the crypto they supply to the protocol and borrowers pay interest to borrow it.One of the most common requests we hear from customers is to be able to buy and sell more cryptocurrencies on Coinbase. We published a process for listing assets, designed in part to accelerate the addition of more cryptocurrencies. We are also investing in new tools to help people understand and explore cryptocurrencies. We launched informational asset pages (see COMP here), as well as a new section of the Coinbase website to answer common questions about crypto.
You can sign up for a Coinbase account here to buy, sell, convert, send, receive, or store COMP today.
Coinbase owns COMP tokens as a result of a 2018 investment in Compound. Coinbase intends to maintain its investment in Compound for the foreseeable future and maintains internal policies that address the timing of permissible disposition of its digital assets, including COMP tokens.
Compound (COMP) is now available on Coinbase was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
-
View all news of Coinbase
Posted on June 18, 2020Scaling connections with Ruby and MongoDB
Coinbase was launched 8 years ago as a Ruby on Rails app using MongoDB as its primary data store. Today, the primary paved-road language at Coinbase is Golang, but we continue to run and maintain the original Rails monolith, deployed at large scale with data stored across many MongoDB clusters.
This blog post outlines some scaling issues connecting from a Rails app to MongoDB, and how a recent change to our database connection management solved some of these issues.
Global VM Lock
At Coinbase we run our Ruby applications using CRuby (aka Ruby MRI). CRuby uses a Global VM Lock (GVL) to synchronize threads so that only a single thread can execute at once. This means a single Ruby process can only ever use a single CPU core at once, whether it runs a single thread or 100 threads.
Coinbase.com runs on machines with a large number of CPU cores. To fully utilize these cores, we spin up many CRuby processes, using a load-balancing parent process that allocates work across these child processes. In the application layer, it’s hard to share database connections between these processes, so instead each process has its own MongoDB connection pool that is shared by that process’ threads. This means each machine has 10–20K of outgoing connections to our MongoDB clusters.
Blue-green deploys
Maintaining product velocity is essential at Coinbase. We deploy to production hundreds of times a day across our fleet. In the general case we use blue-green deployments, spinning up a new set of instances for each deploy, waiting for these instances to report healthy, before shutting down the instances from the previous deploy.
This blue-green deploy approach means we have 2x the count of server instances during these deploys. It also means 2x the count of connections to MongoDB.
Connection storms
The large count of connections from each instance, combined with the amount of instances being created during deploys, leads to our application opening tens of thousands of connections to each MongoDB cluster. Deploying during high traffic periods, when our application is auto-scaled up to handle incoming traffic, we would see spikes of almost 60K connections in a single minute, or 1K per second.
Hoping to reduce some of this connection load on the database, in March we modified our deployment topology, introducing a routing layer designed to transfer this load from the `mongod` core database process to a `mongos` shard router process. Unfortunately the connections were similarly affecting the `mongos` process and didn’t resolve the problem.
We experienced various failure modes from these connection counts, including an unfortunate interaction where the Ruby driver could cause a connection storm on an already degraded database (this has since been fixed). This was seen during a prolonged incident in April, as described in this Post Mortem, where we saw connection attempts above MongoDB’s 128K maximum to a single host.
MongoDB connection attempts to a single cluster, grouped by replica set member Proxying connections
The vast amount of connections from our Rails application is the root problem; we had to focus on reducing these. Analyzing the total time spent querying MongoDB demonstrated these connections went mostly unused; the application could serve the same amount of traffic with 5% of the current connection count. The obvious solution was some form of external connection pooling, similar to PgBouncer for PostgreSQL. While there was prior art, there was no currently supported solution for connection pooling for MongoDB.
We decided to prototype our own MongoDB connection proxy, which we call `mongobetween`. The requirements were simple: small + fast, with minimal complexity and state management. We wanted to avoid having to introduce a new layer in Rails, and didn’t want to reimplement MongoDB’s wire protocol.
`mongobetween` is written in Golang, and is designed to run as a sidecar alongside any application having trouble managing its own MongoDB connection count. It multiplexes the connections from the application across a small connection pool managed by the Golang MongoDB driver. It manages a small amount of state: a MongoDB cursorID -> server map, which it stores in an in-memory LRU cache.
Results
Since rolling out the connection proxy, we’ve dramatically reduced the overall count of outgoing connections to MongoDB, by around 20x. Deploy connection spikes which used to hit 30K now hit 1.5K connections. The application steady state, which used to require 10K connections per MongoDB router, now only needs 200–300 connections total:
MongoDB connections drop significantly May 21st after deploying the proxy Open source
Today we’re announcing that we are open-sourcing the MongoDB connection proxy at github.com/coinbase/mongobetween. We would love to hear from you if you are experiencing similar MongoDB connection storm issues and would like to chat about our solution. If you’re interested in working on challenging availability problems and building the future of the cryptoeconomy, come join us.
This website contains links to third-party websites or other content for information purposes only (“Third-Party Sites”). The Third-Party Sites are not under the control of Coinbase, Inc., and its affiliates (“Coinbase”), and Coinbase is not responsible for the content of any Third-Party Site, including without limitation any link contained in a Third-Party Site, or any changes or updates to a Third-Party Site. Coinbase is not responsible for webcasting or any other form of transmission received from any Third-Party Site. Coinbase is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement, approval or recommendation by Coinbase of the site or any association with its operators.
All images provided herein are by Coinbase. All trademarks are property of their respective owners.
Scaling connections with Ruby and MongoDB was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
-
View all news of Coinbase
Posted on June 18, 2020Compound (COMP) is launching on Coinbase Pro
On Tuesday, June 23, COMP began trading on Coinbase Pro.
On Monday, June 22, we began accepting inbound transfers of COMP to Coinbase Pro. Trading began the following day, June 23.
Once sufficient supply of COMP is established on the platform, trading on our COMP-USD and COMP-BTC order books will start in phases, beginning with post-only mode and proceeding to full trading should our metrics for a healthy market be met.
Compound (COMP) is an Ethereum token that governs the Compound protocol. The protocol allows anyone to supply or borrow Ethereum tokens through a decentralized market. Suppliers earn interest on the crypto they supply to the protocol and borrowers pay interest to borrow it.
Please note that COMP is not yet available on Coinbase.com or via our Consumer mobile apps. We will make a separate announcement if and when this support is added.
The Stages of the COMP Launch
There will be four stages to the launch as outlined below. We will follow each of these stages independently for each new order book. If at any point one of the new order books does not meet our assessment for a healthy and orderly market, we may keep the book in one state for a longer period of time or suspend trading as per our Trading Rules.
We will publish tweets from our Coinbase Pro Twitter account as each order book moves through the following phases:
- Transfer-only. Starting on Monday, June 22, customers will be able to transfer COMP into their Coinbase Pro account. Customers will not yet be able to place orders and no orders will be filled on these order books. Trading will begin on or after 9AM Pacific Time (PT) the following day, if liquidity conditions are met.
- Post-only. In the second stage, customers can post limit orders but there will be no matches (completed orders). Order books will be in post-only mode for a minimum of one minute.
- Limit-only. In the third stage, limit orders will start matching, but customers are unable to submit market orders. Order books will be in limit-only mode for a minimum of ten minutes.
- Full trading. In the final stage, full trading services will be available, including limit, market, and stop orders.
One of the most common requests we receive from customers is to be able to trade more assets on our platform. Per the terms of our listing process, we anticipate supporting more assets that meet our standards over time.
You can sign up for a Coinbase Pro account here to start trading. For more information on trading COMP on Coinbase Pro, visit our support page.
Please note Coinbase owns COMP tokens as a result of a 2018 investment in Compound. Coinbase intends to maintain its investment in Compound for the foreseeable future and maintains internal policies that address the timing of permissible disposition of its digital assets, including COMP tokens.
— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —
This website contains links to third-party websites or other content for information purposes only (“Third-Party Sites”). The Third-Party Sites are not under the control of Coinbase, Inc., and its affiliates (“Coinbase”), and Coinbase is not responsible for the content of any Third-Party Site, including without limitation any link contained in a Third-Party Site, or any changes or updates to a Third-Party Site. Coinbase is not responsible for webcasting or any other form of transmission received from any Third-Party Site. Coinbase is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement, approval or recommendation by Coinbase of the site or any association with its operators.
All images provided herein are by Coinbase.
Compound (COMP) is launching on Coinbase Pro was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
-
View all news of Coinbase
Posted on June 17, 2020Introducing Rosetta: Build once. Integrate your blockchain everywhere.
By Surojit Chatterjee, Chief Product Officer
At Coinbase, we believe a thriving and open ecosystem is necessary for the promise of crypto to be realized, ultimately leading to our vision of more economic freedom for the world. As early as 2016, the Coinbase Secret Master Plan stated our commitment to the open source communities around crypto protocols. And last year, we outlined our strategy of connecting and growing the cryptoeconomy.
Today we’re launching Rosetta, an open-source specification and set of tools that makes integrating with blockchains simpler, faster, and more reliable. This is a step toward our commitment to support an open ecosystem.
Coinbase initially developed Rosetta as the middleware used to integrate blockchains into its platform securely and painlessly, and today we’re sharing it with the world in the hopes it can ease development and integration time for both developers and crypto platforms alike.
The number of blockchains has grown dramatically in recent years, and with it has come a proliferation of different node and wallet APIs that can be challenging to navigate. The goal of Rosetta is to standardize how to interact with blockchains, making it easy for anyone to build on top of a blockchain for a variety of different use cases.
For developers of new blockchain projects, the Rosetta interface makes it easier to ensure compatibility with exchanges that use Rosetta, and can dramatically speed up the time it takes exchanges to integrate with new blockchains and protect customer funds by ensuring specific security conditions are met.
For the broader community of crypto developers, Rosetta makes it easier to build cross-blockchain applications such as block explorers, wallets and dapps. Instead of writing custom parsing for every supported blockchain, applications can use a blockchain project’s Rosetta implementation to read on-chain data and construct transactions in a standard format; minimizing code and simplifying maintenance.
As open source software, Rosetta is easy for anyone to contribute to, and some blockchain teams have already started providing feedback and support. These teams include Filecoin, Celo, Near, Oasis, Coda, Ontology, Kadena, Handshake, Blockstack, and Sia.
A guide to Rosetta, including full documentation, can be found here. The website also includes common development issues that incorporate lessons learned from our years of conversations with blockchain developers and customers. We hope these lessons will be uniquely valuable for new blockchains that want to attract developers, get listed on major exchanges, and build great user experiences.
Over time, our goal is to support a thriving ecosystem of Rosetta interfaces for many more blockchains including Bitcoin & Ethereum, along with even more tools and community resources. Check out our open source guidelines or get started here. If you’d like to learn more, join our Discourse.
This website contains links to third-party websites or other content for information purposes only (“Third-Party Sites”). The Third-Party Sites are not under the control of Coinbase, Inc., and its affiliates (“Coinbase”), and Coinbase is not responsible for the content of any Third-Party Site, including without limitation any link contained in a Third-Party Site, or any changes or updates to a Third-Party Site. Coinbase is not responsible for webcasting or any other form of transmission received from any Third-Party Site. Coinbase is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement, approval or recommendation by Coinbase of the site or any association with its operators.
Implementing the Rosetta tools and/or guidance found on the Rosetta website does not guarantee an asset to be listed on Coinbase. Coinbase evaluates prospective assets against our Digital Asset Framework to assess factors like security, compliance, and the project’s alignment with our mission of creating an open financial system for the world. To apply for listing, fill out an application here.
Coinbase does not endorse, promote, or specify the listing or integration requirements for any third party projects, exchanges, or cryptocurrencies mentioned in this blogpost or otherwise. Any descriptions of functionality and services provided are for information only. Coinbase is not responsible for any loss of funds or other damages caused as a result of using the projects described above.
Unless otherwise noted, all images provided herein are by Coinbase.
Introducing Rosetta: Build once. Integrate your blockchain everywhere. was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
-
View all news of Coinbase
Posted on June 12, 2020Bootstrapping the Coinbase Monorepo
How Coinbase is using a monorepo to provide developers with world-class tooling, security, and reliability
By Luke Demi
Over the past year, the developer productivity team at Coinbase has been focused on delivering a Bazel monorepo to enable engineers to more easily collaborate across the company and position the company on industry standard tooling.
Today, our Bazel monorepo is home to over 100 projects and 22% of Coinbase’s daily deployments across four different languages (Go, Ruby, Node, Python). Our long term goal is to migrate all of the projects within the company to the monorepo. Our current trajectory has us accomplishing this goal by early 2021.
The purpose of this writeup is to:
- Outline our decision making process for choosing to go down the path of monorepo
- Describe some of the generic challenges faced by monorepos
- Describe some of the specific challenges we faced
- Extrapolate on our trajectory
Why monorepo?
Why did Coinbase choose to put a pause on our existing developer productivity tooling and head down the contentious path of monorepo?
Dependency Management
The first major reason was to solve our story around dependency management and code sharing in general. Like many other companies with many smaller services — we frequently create libraries which are shared across many (or all!) services in the company. As we’ve grown, more and more of these types of libraries (think: logging, server boilerplate, authn/authz) have been created and depended on by the nearly thousands of services in our ecosystem.
These code sharing relationships can frequently become complicated, with services depending on other services and libraries depending on other libraries. In a polyrepo world, these relationships are further complicated by the need to version and publish internal services.
As the number of service-to-service to library-to-library relationships grow within a company, so does the need for sophisticated tooling to ensure libraries are up to date and working across the company. Adding an improvement to a critical library could take months to roll out across all of the services within an organization. There’s an accountability gap between the team making the library change and the customer teams who later apprehensively bump their versions.
A monorepo solves these versioning and publishing issues by simply removing the concepts entirely.
By definition, in a monorepo, there is only ever a single version of a given library in the codebase. Any changes made to that library are immediately rolled out to the entire company, and their impact is seen by the result of the monorepo’s test suite after the change.
Additionally, there is no longer the need for sophisticated tooling to manage publishing, importing, and version bumping libraries. Instead, because all code lives together in a single repository, there’s never a need to publish packages to another location. If you want to rely on code written by another team, it’s as easy as importing it into your service. If you want to write a new shared library it’s as easy as writing the code in /shared and depending on it from your service.
As a result of this new workflow, changes made to widely-depended libraries will immediately trigger test suites for any dependent services. After these tests are run, the library owner can feel confident that their change has been made safely to all downstream services without manually testing those services by version bumping or waiting potentially weeks for service owners to update and find incompatibilities.
Industry Standard Tooling
The second major reason for moving forward with the monorepo is our desire to position the company on industry standard tooling.
Our existing tooling at Coinbase relies on internal, proprietary tooling to define the relationship between code and configuration. Developers use a Dockerfile to define the relationship between the code in their repository and the resulting (to be deployed) Docker image(s). These Docker images are then mapped to a docker-compose-like file that maps configuration to the generated Docker images. Finally, a tool called Codeflow handles building and mapping the relationship between these Docker images and their associated configuration before sending off a “release” to our deployment pipelines.
Our challenge with this current strategy is that Codeflow was initially designed with the strong assumption that a single GitHub repository equals a single Docker image equals a single “deployment” per AWS account.
Over time, we’ve expanded (bolted on) Codeflow features to allow multiple Docker images per repository and multiple “configurations” per AWS account. However the end result of these bolt-on improvements was a stiff, proprietary, opaque chain that ties these Github repositories to their resulting multiple Docker images, then further ties them to the multiple “cloud” configurations for each of the AWS accounts (“environments”) which they are deployed.
Even at the end of all this work, engineers still desired the ability to better customize the chain from source code to “release” (Docker images + cloud metadata). Every new Codeflow feature led to an even steeper learning curve for engineers to understand these proprietary relationships.
What we needed was a way to explicitly and flexibility define the relationship between code and the resulting deployable images and metadata.
Bazel provides the building blocks required to give developers unlimited customization and visibility into the ways that their code is deployed. Rather than pre-defining opaque or stiff relationships between source code and a “release”, a typical chain of Bazel “rules” from source code to a deployable “release” might look something like this:
Developers are now given the flexibility to deploy their code with whatever configuration they see fit. The process by which we construct these relationships is now flexible and transparent.
We are investing in monorepo tooling because it allows us to seamlessly share code and more quickly roll out library and security changes across the organization. Additionally, it positions ourselves on industry standard tooling that more clearly defines the relationship between code and what is deployed to our cloud environments.
Generic Monorepo Challenges
Migrating all of your code into a monorepo leads to one obvious and glaring problem: the majority of the tooling released over the past decade has been designed with the assumption that one “git repository” == “one suite of tests” == “a discrete set of built artifacts”. Based on that assumption, in a polyrepo world, any time a change is pushed to a git repository, all builds and tests for that repository are run in continuous integration (CI).
However, in a monorepo, all of an organization’s services and libraries live within the same repository. Based on the above assumption, every commit to the monorepo would require testing every single test and building every single build.
Therein lies a core challenge of operating a monorepo: it no longer makes sense to fully test and build the entire repository on each commit.
Instead, we must rely on tooling like Bazel to organize all services and libraries into an explicit “build dependency graph” that we can leverage to build and test exactly what is required on each commit.
Even with tooling like Bazel to build a dependency graph of Rules (“building blocks”) within a repository, there are inevitably a significant number of Rules/libraries/services that all other libraries and services depend on within the monorepo. So simply cloning down the repository and running the tests or building the outputs of a single project could still require an hour to build.
Getting around this problem in order to bring build and test times to a reasonable level requires that builds be hermetic so that build outputs can be aggressively cached. In the context of a monorepo, “hermetic” means that a build can produce deterministic outputs regardless of which system or time of the day the build is run.
The goal of a deterministic, hermetic build output is that the output of each build can be cached and leveraged for all builds across the company they are produced in CI or a developer’s machine.
The reality is that multiple challenges stand in the way of a truly reproducible, hermetic build. For one, a hermetic build cannot use any part of the host system’s configuration because even just a small change to a system library or OS version across build hosts could result in a complete drop off in cache hit rate. Additionally, every required dependency must either be SHA pinned or have their source code included within the monorepo.
There’s no end to the number of small external influences that can play into the purity of a monorepo’s hermeticity. However, reaching “hermetic enough” to where the majority of builds produce the same outputs can dramatically improve the cache hit rate across build machines and significantly cut build times.
After building a story around caching, we adapted our existing tooling to fetch and update caches between builds to keep build times lower.
Even with a cache, the tooling required to leverage a Bazel monorepo on tooling that was designed with the “git repository” == “one collection of builds/tests” mindset can be frustrating especially as we attempt to present test results for many projects or libraries all within the space designed for single projects.
Monorepo Challenges at Coinbase
At Coinbase, many of the security controls designed to protect sensitive projects are based on the assumption that a single Github repository == a single project and enforced at the Github repository level. Placing every project (along the infrastructure tooling to deploy themselves) within a single repository would then require “one size fits all” security — effectively policing everyone within the monorepo at the level of the most sensitive project in the repository.
At Coinbase, we require consensus (in the form of +1s) for code to be merged to master and deployed to production. The gatekeeper to enforce this behavior is Heimdall — a service we built to keep track of +1s for any given Git commit SHA for a specific Github repository.
Our solution to solve the “one size fits all” security problem in the monorepo was to adapt our internal tool Heimdall to leverage a Github feature called CODEOWNERS. This feature lets you define the individuals or teams that are required to review code through a single CODEOWNERS file at the root of the repository with a special format of `/directory-name @github/team-name`
In our monorepo, we generate the CODEOWNERS file with a script that rolls up the content of OWNER files that live at the root of every directory in the repository. Each OWNERS file simply contains a space separated list of teams whose reviews should be required for a given directory.
We allow for a special syntax next to teams in OWNERS files to signify the total number of reviews required by a specific team so that we can require multiple reviews for certain directories. In the screenshot above you can see the {0} indicating that zero reviews are required for a team, while {2} would require two reviews from that team. Teams without {} syntax are assumed to require one review.
Heimdall parses all of this information with a special syntax in the CODEOWNERS file to ensure that total required reviews are met before a commit can be successfully deployed. While Github will mark a commit as reviewed with just one team approval, we have separate Heimdall GitHub statuses to ensure PRs aren’t accidentally merged without enough commits from the owning teams.
What’s Next?
Our infrastructure team’s goal is to provide developers at Coinbase with world-class tooling, security, and reliability.
While we’ve made great strides towards providing the base layer for the future of world-class tooling, there are still gaps that we need to fill before we can roll the monorepo out to the entire company.
Currently, developers use their own laptops to build and test locally. Unfortunately this means we are unable to leverage the shared caches that we populate in our build and test fleets, leading to long and potentially frustrating local build times (and loud fans!). Our solution is to provide every developer with their own remote Linux machine in EC2 so that they can easily sync up their changes to build and test quickly by leveraging shared caches.
While we’ve added support for all of the major backend languages across Coinbase, our monorepo (and Bazel itself) does not yet have strong support for client teams leveraging React and React Native. We will be working hard to add support for these frontend languages over the balance of the year.
Finally, while our current build and test tooling has scaled to ~100 projects, we expect the monorepo to expand in projects and lines of code by over an order of magnitude during the next year. We expect that this growth will cause us to run up against scaling limitations in our current caching, version control, and build strategies.
Overall, that’s the high level overview of the promise, challenges, and future of the Coinbase monorepo — expect more blog posts discussing in-depth components of our monorepo over the coming months.
If you are interested in solving complex technical challenges like this, Coinbase is hiring.
This website contains links to third-party websites or other content for information purposes only (“Third-Party Sites”). The Third-Party Sites are not under the control of Coinbase, Inc., and its affiliates (“Coinbase”), and Coinbase is not responsible for the content of any Third-Party Site, including without limitation any link contained in a Third-Party Site, or any changes or updates to a Third-Party Site. Coinbase is not responsible for webcasting or any other form of transmission received from any Third-Party Site. Coinbase is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement, approval or recommendation by Coinbase of the site or any association with its operators.
Unless otherwise noted, all images provided herein are by Coinbase.
Bootstrapping the Coinbase Monorepo was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
-
View all news of Coinbase
Posted on June 11, 2020Maker (MKR) is now available on Coinbase
Starting today, Coinbase supports Maker (MKR) at Coinbase.com and in the Coinbase Android and iOS apps. Coinbase customers can now buy, sell, convert, send, receive, or store MKR. MKR will be available in all Coinbase-supported regions, with the exception of New York State.
Maker (MKR)
Maker is an ERC20 token that describes itself as “a utility token, governance token, and recapitalization resource of the Maker system.” The purpose of the Maker system is to manage another Ethereum token, the DAI stablecoin, that seeks to trade on exchanges at a value of exactly US$1.00.
One of the most common requests we hear from customers is to be able to buy and sell more cryptocurrencies on Coinbase. We announced a new process for listing assets, designed in part to accelerate the addition of more cryptocurrencies. We are also investing in new tools to help people understand and explore cryptocurrencies. We launched informational asset pages (see MKR here), as well as a new section of the Coinbase website to answer common questions about crypto.
You can sign up for a Coinbase account here to buy, sell, convert, send, receive, or store MKR today.
Maker (MKR) is now available on Coinbase was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
-
View all news of Coinbase
Posted on June 10, 2020Coinbase continues to explore support for new digital assets
We are continuing to explore the addition of new assets for customers around the world.
Coinbase’s goal is to offer support for all assets that meet our technical standards and which comply with applicable laws. Over time we expect our customers around the world will have access through Coinbase platforms to at least 90% of the aggregate market cap of all digital assets in circulation.
We will continue to evaluate prospective assets against our Digital Asset Framework to assess factors like security, compliance, and the project’s alignment with our mission of creating an open financial system for the world. To apply for listing, fill out an application here.
Digital Assets Under Review
Today we’re announcing that we are exploring the addition of a range of new assets. As part of the exploratory process customers may see public-facing APIs and other signs that we are conducting engineering work to potentially support these assets.
These new assets include, in alphabetical order: Aave, Aragon, Arweave, Bancor, COMP, DigiByte, Horizen, Livepeer, NuCypher, Numeraire, KEEP Network, Origin Protocol, Ren, Render Token, Siacoin*, SKALE Network, Synthetix, and VeChain.
*Note that we are not currently exploring support for Siafunds.
Our decision to support any asset requires significant technical and compliance review and may be subject to regulatory approval in some jurisdictions. We therefore cannot guarantee whether or when any above-listed asset will be listed on a Coinbase product in any jurisdiction. As per our listing process, we will add new assets on a jurisdiction-by-jurisdiction basis, subject to applicable review and authorizations. The omission of assets from this publication does not disqualify any such asset from active review and potential listing.
Our customers can expect Coinbase to make future, similar announcements as we continue to explore the addition of numerous assets across the platform.
This website contains links to third-party websites or other content for information purposes only (“Third-Party Sites”). The Third-Party Sites are not under the control of Coinbase, Inc., and its affiliates (“Coinbase”), and Coinbase is not responsible for the content of any Third-Party Site, including without limitation any link contained in a Third-Party Site, or any changes or updates to a Third-Party Site. Coinbase is not responsible for webcasting or any other form of transmission received from any Third-Party Site. Coinbase is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement, approval or recommendation by Coinbase of the site or any association with its operators.
All images provided herein are by Coinbase. All trademarks are property of their respective owners.
Coinbase continues to explore support for new digital assets was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
-
View all news of Coinbase
Posted on June 8, 2020Container technologies at Coinbase
Why Kubernetes is not part of our stack
By Drew Rothstein, Director of Engineering
TLDR: Container orchestration platforms are complex and amazing technologies, helping some businesses and teams solve a whole suite of problems. What’s commonly overlooked however, is that container technologies also create a large set of challenges that must be overcome to prevent failures.
This post is adapted from an internal blog post as I haven’t seen many write-ups like this externally available. Minimal redaction has been done and images have been added to provide more flare. If you are interested in working on some of what we discuss below — we are actively hiring on our Infrastructure team.
History
Before jumping into the current day, it is important to understand the technologies that led us here.
- 1980s: chroot
- 1990s: jail
- 2000s (early): jail > FreeBSD
- 2000s (mid): cgroups, 2.6.24
- 2000s (late): LXC (Linux Containers), 2.6.4
- 2010s (early): Docker
- 2010s (late): Kubernetes
There is a more detailed history in Chapter 7 of Enterprise Docker if interested.
Without containers as we know them today, let’s go back ~10yrs. At this time we did not have/use docker, rkt, or any other mainstream containerized wrapper/service. Most large-scale companies built in-house systems to bundle their applications to go from source code to deployment in production. What engineers ran on their machine was usually not what was running in production or if it was, it was lovingly one-off built/packaged in a manner that was likely very custom and complex.
In this world of an in-house system to bundle and deploy applications there was a large operations team, usually in a platform or infrastructure organization that would manage the bundle/building processes, deployment, and post-deployment. These roles were generally highly operational involving troubleshooting bad hosts, diagnosing specific dependency issues on OS patches/upgrades, etc. Post-deployment had minimal to no automated orchestration and involved capacity planning, ordering more servers, getting them racked/installed, and somehow getting software updated on them.
If you were lucky, there was some regular process to build a “golden image” (think: Packer by Hashicorp) that was well documented, potentially even codified, and run by a Continuous Integration system such as Hudson (previous to Jenkins {ref}). These images were somehow distributed to your systems either manually or automatically through some sort-of configuration management utilities and then started in some ordering, likely with parallel SSH or similar.
This past decade everything has changed. We went from gigantic monolithic applications to breaking down services into more discrete and less coupled parts. We went from having to build/own your own compute to having a managed or Public Cloud offering with a couple clicks and a credit card. We went from scaling applications vertically to re-architecting them to scale horizontally. All of this was happening at the same time that societal changes were also occurring: cell phones in every pocket, network speeds improving, network latencies dropping across the world, to doing everything online from booking your dog walker to commoditized video conferencing.
AWS’s offering in 2009 was quite limited. For perspective, it wasn’t until 2008 when AWS’s EC2 offering exited beta and began offering an SLA (ref). For reference, GCP didn’t launch a compute offering in GA until 2013 (ref).
Why do companies choose to containerize their applications?
Companies choose to containerize their applications to increase engineering output/developer productivity in a quick, safe, and reliable manner. Containerizing is a choice made vs. building images, although containers can sometimes be built into images, but that is out of scope (ref).
Containers enable engineers to develop, test, and run their applications locally in the same or similar manner that they will run in other environments (staging and production). Containers enable bundling of dependencies to be articulated and explicit vs. implied (the OS will always contain package $foo that my service depends on). Containers allow for more discreet service encapsulation and resource definition (using X CPUs and Y GB of Memory). Containers inherently enable you to think about scaling your application horizontally vs. vertically, resulting in more robust architectural decisions.
Some of these points could be argued in great detail. These are purposely bold and a bit over-extended to move the conversation forward as this isn’t a discussion of the pros/cons of containerization or service-ification (i.e. the breakdown of monolithic applications to a proliferation of more discreet services that run separately).
What about virtualization?
Virtualization is the concept of being able to run multiple containers on an OS virtualized system (ref). Containers can only see the devices/resources granted to it. On a managed compute platform such as AWS you are actually running below a Hypervisor (ref) which manages the VMs that your OS and resulting containers run within.
Simplified diagram Virtualization enables the world of containers today. Without the ability to virtualize, hardware resources running multiple applications in containers wouldn’t be possible today.
What problem does a container orchestration platform (Mesos, Kubernetes, Docker Swarm) solve?
A container orchestration platform solves the following types of problems:
- Managed/Standardized deployment tooling (deployment).
- Scaling of applications based-on some defined heuristic (horizontal scaling).
- Re-scheduling/Moving containers when failures occur (self-healing).
While some platforms may state that they have other features such as storage orchestration, secret/config. management, and automatic bin packing to name a few: the reality is that these generally do not work for larger scale installations without intense investments either in forking / customization or through integrations and separation.
For example, most folks that run large-scale container orchestration platforms cannot utilize their built-in secret or configuration management. These primitives are generally not meant, designed, or built for hundreds of engineers on tens of teams and generally do not include the necessary controls to be able to sanely manage, own, and operate their applications. It is extremely common for folks to separate their secret and config. management to a system that has stronger guarantees and controls (not to mention scaling).
Similarly for service discovery and load balancing it is quite common to separate this out and run an overlay or abstract control plane. It is quite common to deploy Istio to handle this for Kubernetes. Managing and running Istio is not a trivial task and many modern-day cluster outages are due to misconfiguration of this control plane/service mesh and a lack of understanding of the minute details of it.
What do we use as our container orchestration platform?
Our container orchestration platform is Odin + AWS ASGs (auto-scaling groups). When you click Deploy from Codeflow (our internal UI for deployments), Odin is invoked with an API call from Codeflow. Odin kicks off a step function and begins to deploy your application. New VMs are stood up in AWS and loaded into a new ASG, your software is fetched from various internal locations, a load balancer starts health-checking these new instances, and eventually traffic is cut over in a Blue/Green manner to the new hosts in the new ASG behind the load balancer.
Our container orchestration platform is extremely simple (on purpose). We enable the same key features of Kubernetes: A single Deploy + Rollback button in Codeflow, Scaling based-on some defined heuristic (we support custom AWS metrics or standard CPU metrics), and re-scheduling/moving of your containers if your VM dies/becomes unhealthy in your ASG.
To handle secrets and configuration management we have built a dynamic configuration service that provides libraries to all internal customers with a p95 of 6ms. It is backed by DynamoDB and serves 100s of thousands of requests per minute of synchronous and asynchronous methods types.
To handle service discovery and load balancing we utilize Route53 (DNS), ALBs (Application Load Balancers), and client-side load balancing for gRPC either natively or through Envoy. We expect to invest more here later in the year.
Why do we not run Kubernetes?
Running Kubernetes does not solve any customer (engineering) problems. Running Kubernetes would actually create a whole new set of problems.
- We would need to build/staff a full-time Compute team. While we may do this anyway as we grow, this would be required immediately so that they could focus on building out tens of clusters (likely separate for each team/org), starting to scope/build the wrapping/glue tooling, starting to build out the abstract control plane/service mesh, etc.
- Securing Kubernetes is not a trivial, easy, or well understood operation. To enable us to own/operate Kubernetes we would need the same tooling and controls that we have today with our entire platform (Odin, ASGs, Step Deployers — and everything they enforce). To build these same primitives providing the same level of safety that these provide today would be a substantial investment both by a (future) Compute team and our Security team.
- Managed Kubernetes (EKS on AWS, GKE on Google) is very much in its infancy and doesn’t solve most of the challenges with owning/operating Kubernetes (if anything it makes them more difficult at this time). At AWS they are scaling their support/operations teams to run EKS and at Google it isn’t uncommon for them to have multi-hour outages with GKE (ref). You are trading off some operations issues and challenges to another operations team (and removing a lot of visibility).
- Cluster upgrades and management require a much more operationally heavy focus than we have today. The only way to sanely run Kubernetes is by giving teams/orgs their own clusters (similar to giving them their own AWS accounts or GCP projects). Upgrading clusters and patching vulnerabilities is not a quick/easy task even with Istio and associated tooling. Generally you have to build/run a secondary cluster, failover all applications, and then fail back after an upgrade. This primitive is not built into any abstracts at this point in time. While this may exist for managed clusters (GKE) it doesn’t always work as you might expect and rolling back once started is generally not well handled.
- Today, we do not carry this burden. We run on a hardened OS with minimal > no dependencies. Our AMI rollout is managed starting with development and then moving forward after weeks of testing. If we need to rollback we have the ability to do so with a trivial one-line change. On average we spend < 5hrs/month on anything even closely related to this area of concern.
Additional references on the complexities of owning/operating Kubernetes & Istio:
- OpenStack (Kubernetes Issues At Scale 900 Minions)
- OpenAI (Scaling Kubernetes to 2,500 Nodes)
- Civis (Breaking Kubernetes: How We Broke and Fixed our K8s Cluster)
- k8s.af
Securing Kubernetes
Let’s discuss some of the complexity of securing and running Kubernetes as a business that stores more than $8 billion in crypto assets.
Components
The basics of securing a Kubernetes cluster (ref) are well known/understood but once you dig into each of them the complexities start to unravel. Securing all of the system components (etcd, kubelet), the API server, and any abstracts/overlays (Istio) opens up a lot of surface to understand, test, and secure. Going deep into namespaces, seccomp, SELinux, cgroups, etc. is all required given the increased attack surface. Kubernetes is so large that it has its own CIS benchmark & InSpec suite (thankfully).
Vulnerabilities
A small list of references that provide a good starting point for researching:
- CVE-2019–5736 (8.6 High): Allows attackers to overwrite the host runc binary (and consequently obtain host root access).
- CVE-2019–11246 (6.5 Medium): If the tar binary in the container is malicious, it could run any code and output unexpected, malicious results.
- CVE-2019–11253 (7.5 High): Allows authorized users to send malicious YAML or JSON payloads, causing the API server to consume excessive CPU or memory, potentially crashing and becoming unavailable.
Overview
Kubernetes is a powerful PaaS as a kit with lots of security-relevant options to support the variety of deployment scenarios that it can be used in. It’s incredibly valuable from a security perspective when it is the universal consensus choice for PaaS, because most of those options can be abstracted away, and secondary systems must be put into place to support its use.
Kubernetes is fundamentally designed for workload orchestration — Trust is not the differentiator or purpose behind the encapsulation or pieces in Kubernetes; The multi-tenancy purpose is for bin packing and not in support of furthering permission boundaries. It provides several layers at which you can choose to place mild boundaries of varying enforceability. Some of these boundaries are built-in, while others are simply integration points for other tools to help manage. Here are some of the primitives Kube provides (and doesn’t provide) to isolate workloads.
Control Plane (AWS Account / GCP Project)
Kubernetes clusters operate within the services and networks that are provided to them, and naturally have some interaction with the AWS/GCP control plane such as provisioning load balancers for ingress, accessing secrets stored in KMS, etc. Teams grow and expand to have separate accounts, projects, and further isolation over time. A separate AWS account or GCP project is the primary primitive by which you can achieve total IAM segmentation.
A Kubernetes cluster, on the other hand, needs to operate within one AWS account (even if federated with other clusters elsewhere). This limits segmentation options and flexibility. We could provision a cluster per team or service, but this takes away many of the efficiency gains that Kubernetes brings, and brings on new management problems, like meta-orchestrating all of those clusters.
Clusters & Nodes & Pods & Containers (Oh my!)
Clusters
Cluster master (API) servers are a secondary control plane (besides the AWS one) that we need to secure as well. Service accounts and access scopes, which containers can assume to access resources both within and outside the cluster, are just as complex as AWS’ IAM is, and need to be mapped against one another strictly so that a breakout does not affect the, e.g., AWS control plane.
Nodes
The operating system of the underlying nodes must be maintained much as we do today. In fact, our OS is very similar to the base OS Google uses for GKE. While we wouldn’t necessarily have to change anything to move our OS to Kubernetes, we wouldn’t gain anything either.
Pods
Creation of pods in the cluster, and the rules about what standards they have to meet to be created, are accomplished through PodSecurityPolicy, which operates similarly to Salus and our consensus management tooling today. We would have to invest in significant integration work, and additional open source dependencies, to cleanly integrate them.
Pods are segmented from each other through networking policies, much as we do today with Security Groups and/or our internal service framework. But in the world of Kubernetes, identity, authentication, and authorization of pods to communicate with each other involve a number of supporting technologies, such as SPIFFE and SPIRE for identity format and attestation below the node level, Envoy for authorization gating, Istio for authN and Z orchestration, and OPA for authorization policy. Each of these is a significant effort to standardize and adopt.
Containers
Containers are not security boundaries, they’re resource boundaries. In order to define security boundaries around containers, you need to delve into custom kernel namespaces, syscall filtering, mandatory access control frameworks, and/or vm-based isolation technologies designed for containers like gVisor.
Currently, we have not invested much in this area because we do not operate in a multi-tenant fashion. If we moved to a multi-tenant model, we would have to make significant investments here almost immediately so that we could trust that pods/containers are only running on the same nodes as similarly classified pods, and that they are not interfering with one another with host/vm isolation technologies.
When will we run Kubernetes and is Kubernetes in our future?
If/when there are significant use cases for a more advanced container orchestration platform we will likely first visit the problem statement. If this is something that can easily be added to our existing platform: we will likely visit that first and explore/scope from there. If we deem it unreasonable to extend/add to our platform then we would visit all potential options — not just Kubernetes. It is much more likely that we would visit AWS’s managed offerings first such as Fargate and ECS before diving in to Kubernetes based on the above.
If/when there are significant gains to be had by our engineers by offering Kubernetes (or any other container orchestration platform) we will explore offering them. At this time there is not a significant gain to be had by offering Kubernetes. This may change if/when Kubernetes offers enough new features that we haven’t kept up, they have paid down their technical debt (or we have not), or our customers require new functionality that they are able to offer and we are not in the foreseeable future. If the barrier to entry of our current platform were to significantly change and that were now a clear differentiator, then we would also explore offering a different platform.
If/when we hit the limits of our existing platform, are too deeply burdened or foresee being too deeply burdened in our platform due to missing features that our customers need, extending our platform is becoming too onerous, or we are having too many outages that are violating our SLA, than we would likely revisit a different container orchestration platform.
If/when we lose support of a major upstream dependency such as AWS or ASGs, we would then look into other options.
These are a few of the reasons we might choose to look into another container orchestration platform. At this time we have no plans to build/own/operate Kubernetes.
Doesn’t Kubernetes solve various problems such as re-balancing/auto-healing, auto-scaling, and service discovery? How do we solve these today?
Kubernetes at a smaller scale solves most of these problems without a lot of fuss. At a larger scale it requires a lot more thought, glue code, and putting wrappers / safe guards on pretty much everything to make it work safely and reliably. Generally, as mentioned above, folks tend to add a Service Mesh such as Istio to enable more advanced features / requirements.
Today we solve:
- Re-balancing/auto-healing with Odin and ASGs.
- Service discovery with DNS and Envoy.
Kubernetes has Storage Orchestration and we don’t have that today, should we?
We have two major stateful applications at Coinbase today- blockchain nodes and the trading engines that could be potential customers of a feature such as storage orchestration. For the former (blockchain nodes) the usage of storage is fairly custom and we have built a custom deployer that gives them the features that they need. For the latter (trading engines), we embedded from the Reliability (SRE) team and provided support to a number of their specific challenges.
While having Storage Orchestration built-in to Kubernetes might have been a nice starting point for both blockchain nodes and the trading engines- a lot of the same issues we have with the underlying technology would still exist.
What is the future of a Container Orchestration Platform if not Kubernetes?
We will explore and migrate to a higher-level abstracted service for some applications. We will explore Fargate and ECS as contenders for this purpose. The current initial reasons would be utilization and cost improvements — both of which are not very customer focused. We may choose to wait until we have more customer focused reasons to implement.
Potential customer focused asks would be around deployment times, deployment patterns (beyond canaries), more complex service mesh needs than exist today, or specific improvements/ features that may be added to Fargate or ECS that building onto existing tooling is not possible/not reasonable. These are some of the potential customer focused asks that are possible but not known or realized at this time.
Ideally the move to a different underlying container technology would be fairly invisible as the tools to interact with them wouldn’t fundamentally change. The reality of moving to a different platform would likely uncover hidden or unknown expectations about the existing system. How you deploy and debug services in staging and production would still be abstracted, but there may be different features offered that do not exist today.
Do I/we hate Kubernetes? Does Kubernetes fail as a container platform?
No. It is a great tool despite its challenges. Kubernetes has moved our industry forward in an increasingly positive direction. With Kubernetes well into a v1, the development of Knative, Fargate, and Cloud Run are increasingly raising the level of abstraction and solving the underlying challenges with managing Kubernetes. The future is bright. As these underlying challenges are solved, many existing concerns will likely be alleviated in the future.
If you are interested in working on our next generation of container technologies, our dynamic configuration service or other technologies mentioned above — we are actively hiring on our Infrastructure team. Please reach out and we would love to chat with you.
This website contains links to third-party websites or other content for information purposes only (“Third-Party Sites”). The Third-Party Sites are not under the control of Coinbase, Inc., and its affiliates (“Coinbase”), and Coinbase is not responsible for the content of any Third-Party Site, including without limitation any link contained in a Third-Party Site, or any changes or updates to a Third-Party Site. Coinbase is not responsible for webcasting or any other form of transmission received from any Third-Party Site. Coinbase is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement, approval or recommendation by Coinbase of the site or any association with its operators.
Unless otherwise noted, all images provided herein are by Coinbase.
Container technologies at Coinbase was originally published in The Coinbase Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
