I have been having some trouble with my displacement maps in more then one way. But I think I have found a good solution. What I do now is to have different displacement maps for different level of detail. Large parts, details, and fine details. Connecting them with a Average node with a Multiple/divider node between each displacement map. This makes it really easy to only adjust the shape of a specific level of detail and controlling each map separate rather then one map. This was actually a very simple and easy solution.