Random numbers have great application value in the fields of secure communications, which are commonly used as secret keys to encrypt the information. To guarantee that the information is absolutely secure in the current high-speed communication, the applied random keys should possess a generation speed not less than the encrypted data rate, according to one-time pad theory found by Shannon (Shannon C E 1949 Bell.Syst.Tech.J. 28 656) Pseudo-random numbers generated by algorithm may easily reach a fast speed, but a certain periodicity makes them difficult to meet the aforementioned demand of information security. Utilizing physical stochastic phenomena can provide reliable random numbers, called physical random number generators (RNGs). However, limited by the bandwidth of the conventional physical sources such as electronic noise, frequency jitter of oscillator and quantum randomness, the traditional physical RNG has a generation speed at a level of Mb/s typically. Therefore, real-time and ultrafast physical random number generation is urgently required from the view of absolute security for high-speed communication today. With the advent of wideband photonic entropy sources, in recent years lots of schemes for high-speed random number generation are proposed. Among them, chaotic laser has received great attention due to its ultra-wide bandwidth and large random fluctuation of intensity. The real-time speed of physical RNG based on chaotic laser is now limited under 5 Gb/s, although the reported RNG claims that an ultrafast speed of Tb/s is possible in theory. The main issues that restrict the real-time speed of RNG based on chaotic laser are from two aspects. The first aspect is electrical jitter bottleneck confronted by the electrical analog-to-digital converter (ADC). Specifically, most of the methods of extracting random numbers are first to convert the chaotic laser into an electrical signal by a photo-detector, then use an electrical ADC driven by radio frequency (RF) clock to sample and quantify the chaotic signal in electronic domain. Unfortunately, the response rate of ADC is below Gb/s restricted by the aperture jitter (several picoseconds) of RF clock in the sample and hold circuit. The second aspect comes from the complex post-processes, which are fundamental in current RNG techniques to realize a good randomness. The strict synchronization among post-processing components (e.g., XOR gates, memory buffers, high-order difference) is controlled by an RF clock. Similarly, it is also an insurmountable obstacle to achieve an accurate synchronization due to the electronic jitter of the RF clock. In this paper, we propose a method of ultrafast multi-bit physical RNG based on chaotic laser without any post-process. In this method, a train of optical pulses generated by a GHz mode-locked laser with low temporal jitter at a level of fs is used as an optical sampling clock. The chaotic laser is sampled in the optical domain through a low switching energy and high-linearity terahertz optical asymmetric demultiplexer (TOAD) sampler, which is a fiber loop with an asymmetrical nonlinear semiconductor optical amplifier. Then, the peak amplitude of each sampled chaotic pulse is digitized by a multi-bit comparator (i.e., a multi-bit ADC without sample and hold circuit) and converted into random numbers directly. Specifically, a proof-of-principle experiment is executed to demonstrate the aforementioned proposed method. In this experiment, an optical feedback chaotic laser is used, which has a bandwidth of 6 GHz. Through setting a sampling rate to be 5 GSa/s and selecting 4 LSBs outputs of the 8-bit comparator, 20 Gb/s (=5 GSa/s4 LSBs) physical random number sequences are obtained. Considering the ultrafast response rate of TOAD sampler, the speed of random numbers generated by this method has the potential to reach several hundreds of Gb/s as long as the used chaotic laser has a sufficient bandwidth.