Completely remove vanilla Wordpress tags and categories

functions.php

//Remove categories and tags
  function unregister_categories() {
    register_taxonomy( 'category', array(), array('show_in_nav_menus' => false) );
  }
  add_action( 'init', 'unregister_categories' );
  unregister_widget( 'WP_Widget_Categories' );

  function unregister_tags(){
    register_taxonomy('post_tag', array(), array('show_in_nav_menus' => false));
  }
  add_action('init', 'unregister_tags');

  function remove_tagmenu(){
    remove_menu_page('edit-tags.php?taxonomy=post_tag'); // Post tags
  }
  add_action( 'admin_menu', 'remove_tagmenu' );

Theres an issue, if you test a url with /category/fake-category/ it will give all the posts instead of 404 page