There are three things to consider: settings, operation and connection.
Please check each in the following sequence:

1) Confirm the settings on the Serato DJ Pro screen
1. Check if the [Error: Failed to connect audio!!] or [Driver Not Installed] message shows at the bottom of the screen.
If it is shown, see XDJ-RX3 FAQ > [Troubleshooting - Serato DJ Pro > "The message [Error: Failed to connect audio!!] is shown, or the virtual deck disappears immediately on the Serato DJ Pro screen."]
2. Click the icon in [My Serato] in the upper right area of the screen.
3. Check that [Serato DJ Pro is hardware activated.] and [XDJ-RX3] are shown on the screen.
If they aren't shown, see XDJ-RX3 FAQ >[Troubleshooting - Serato DJ Pro > "The Serato DJ Pro screen doesn't react to anything I do on the XDJ-RX3 when it's connected."]

2) Check the volume level operation
1. Play a track.
2. Set the TRIM knob to the center (12 o’clock).
3. Slide the channel fader to the top.
4. Slide the crossfader to the center.
5. Turn the MASTER LEVEL knob to 2 o'clock.
If the MASTER LEVEL indicator lights up more than half way, there’s no problem in the settings and operation.
If you don’t hear any sound from the headphones at this point, check the following:
6. Turn the HEADPHONES MIX knob to the center.
7. Turn the HEADPHONES LEVEL knob all the way to the left.
8. Press the MASTER CUE button.
9. Turn the HEADPHONES LEVEL knob slowly to the right.

3) Check the connection of speakers and headphones
For details, see the Quick Start Guide.
Even if you’ve already connected them, disconnect and reconnect them to make sure they’re connected properly.
*It's not possible to output audio from PC/Mac speakers.

You can read the Quick Start Guide [here].

The installation of the driver software might be incomplete or the connection might be causing a problem.
If you've already installed the driver software, try again in the following sequence:

■ Windows
1. Close Serato DJ Pro and disconnect the XDJ-RX3 from your computer.
2. Open the control panel (or [Start] > [Settings] > [Apps]) and uninstall [XDJ-RX3 Driver].
3. Quit Windows.
4. Disconnect other USB devices, USB hubs, etc.
5. Start Windows.
6. Download driver software from [here].
7. Install the driver software again.
　*An error will occur when the license agreement screen is displayed and you try to proceed without clicking [I agree].
8. Connect the XDJ-RX3 to another port without using a USB hub, etc.

If you still experience the problem, it might be caused by operating system sound settings, etc.
Remove all other USB devices from your computer.
Follow the instructions in the troubleshooting guide Case 3.1 [Check the default device selected in the [Sound] panel] or Case 6 [here]
*The guide is for rekordbox users but you can fix the problem in the same way.

■ Mac
1. Close Serato DJ Pro and disconnect the XDJ-RX3 from your computer.
2. Download the driver software from [here].
3. Double-click [XDJ-RX3 Uninstaller] to uninstall it.
4. Shut down your Mac and disconnect other USB devices, USB hubs, etc.
5. Start your Mac.
6. Double-click [XDJ-RX3 AudioDriver.pkg] to reinstall it.
7. Connect the XDJ-RX3 to another port without using a USB hub, etc.
8. Follow the instructions in the troubleshooting guide
macOS Big Sur: [here]
macOS Catalina, Mojave, High Sierra: [here]

　
If you still experience the problem, it might be caused by operating system sound settings, etc.
Remove all other USB devices from your Mac, uninstall the virtual audio device etc, and restart the Mac.

Serato DJ Pro or your operating system might not recognize the unit as a controller.

■ Mac
Follow the instructions in the troubleshooting guide [here].

■ Windows
See Case 3 in the guide [here].

You might solve the issue by doing the following:
- Disconnect any USB hubs and connect the unit directly to your PC/Mac.
- Disconnect any other USB devices.
- Quit all other applications.
- Connect to a different USB port.

If the volume level is too high, the sound might be affected by distortion or clipping.
Play a track and check the following:

1. Check the channel level indicator.
Adjust the level with the TRIM knob so the orange color lights up in the louder parts of the track.
2. Check the MASTER LEVEL indicator.
With the channel fader at the top, adjust the level with the MASTER LEVEL knob so the orange color lights up in the louder parts of the track.
3. Turn off Key Lock.

If distortion and sound clipping doesn’t improve, the output level might exceed the allowed input level of the connected speaker or amplifier.
In this case, you need to lower the output level with the MASTER attenuator. For details, see [Changing the settings] and [MIXER category] in the [Utility settings] in the Operating Instructions.

If sound skipping, sound cutting, or noise occurs, see [Troubleshooting - Serato DJ Pro >"Sound skips, cuts, or is affected by noise."]

You can read the Operating Instructions [here].

Three main factors can cause this and there are three possible countermeasures:

■When buffer size is the cause
If the buffer size is too low, sound cutting or noise might occur.
You can adjust buffer size in Serato DJ Pro [Setup] < [Audio] < [USB Buffer size].
First, select a large value such as 20 ms. If the problem clears up, adjust to a value small enough that you don't experience a delay (latency).
*The appropriate value depends on the system specification.

■When the connection or other equipment is the cause
An unstable connection with your PC/Mac may cause these problems.
To solve this, shut down your PC/Mac and disconnect all other USB devices and USB hubs.
Then, connect the XDJ-RX3 directly to a different USB port.

■When your system (PC/Mac) is the cause
Doing the following may solve the issue:
- Quit all other applications.
- Disconnect from the internet and exit security software.
- Turn off wireless connections such as Wi-Fi or Bluetooth.
- Turn off power-saving settings.

If your computer's system doesn’t meet the requirements of Serato DJ Pro, it may cause the issue.
Read details on system requirements [serato.com].

If you can't play audio files with Serato DJ Pro, check if the following applies:

• The file format is one that's not supported by Serato DJ Pro.
• Only certain applications can play the audio file: e.g. the audio file was downloaded from Apple Music, etc.
• The location of the audio file has changed.
• The music file is damaged.

For details, see the Serato DJ Pro Manual, available [serato.com].

When multiple ASIO drivers are installed or multiple audio devices are connected to your PC/Mac, it might take a long time for Serato DJ Pro to recognize the XDJ-RX3.

Disconnect audio devices other than the XDJ-RX3 and uninstall unnecessary ASIO drivers and virtual audio devices. Visit the website of the relevant manufacturers for details of how to do this.