This is not actually the case, the default is internal class/private members, but public seems to be the de-facto choice by a lot of developers.
If you're creating a public class - you have a big responsibility. Your code can now be used by anything else in the application, or even outside it. This means if you change the internal working of it, you will affect everything that uses it - and you have no control over what that is. This means a couple of things.
Naming is super importantNaming of public classes and methods and their expectations should be crystal clear. Don't rely on comments to convey intentions. It must be highly obvious what the class is for, what it does, and how method parameters are used. It should be specific and adhere to the Single Responsibility Principle - one reason to change. If someone comes along and adds a parameter to one of your methods to get it to do something else, this is an indication that the class's purpose was not clear enough.
If the naming is not clear, the purpose gets diluted, and the class will be modified away from its original intentions. This causes code to bloat and become more prone to bugs.
Tip: This is yet another reason to avoid "generic static helpers"!
Tests are the documentation of a contractIf your classes or methods are public, tests need to cover every possible way that the code can be used - every single path. What this means is that anyone using the code will know exactly what is expected of it because its functionality is documented by tests. Any usage of the code will already be covered by a test.
If you don't create tests, anyone can come along and change this code which can then affect any number of consumers.
Don't make life hard for yourselfDon't want this responsibility? Simple: Don't make your classes and methods public. Of course internal members should always be tested and code should be named well. But these issues take on less importance if the code is isolated.
Go with the lowest access first and only open it up as and when you need to, When you do open it up, make sure that it does what it says it will, and it is fully tested. The important thing to remember is that when a class is public, it increases its exposure, which allows it to be coupled with more code. This is a significant cause of spaghetti code and regression issues as the code is maintained.